Analisa dan Perancangan Machine Learning Untuk Mendeteksi Kegagalan Job di Apache Spark

Authors

  • Eri Dariato Universitas Dian Nusantara, Jakarta, Indonesia, Indonesia

DOI:

https://doi.org/10.29240/arcitech.v2i1.4124

Keywords:

Database, Artificial Intelligence, Apache Spark, Feature Engineering

Abstract

A collection of data stored in a database, so the longer the data, the bigger the data, because the data processed is very large, processing time in Apache Spark can take up to a dozen or tens of hours. Sometimes, the Apache Spark application even fails. Therefore, to minimize the waiting time that could have been avoided or reduced, artificial intelligence through Machine Learning will be used to detect whether an Apache Spark application will fail or run smoothly. Factors to determine this failure are called features and are generated through the feature engineering process. The purpose of this research is to design Machine Learning so that it is able to find out what features will determine the success or failure of the Apache Spark application. The research method used is the Prototyping process model.

Downloads

Download data is not yet available.

Author Biography

Eri Dariato, Universitas Dian Nusantara, Jakarta, Indonesia

Fakultas Teknik, Prodi Teknik Informatika

References

Apache Hadoop. (2022). Diambil kembali dari http://hadoop.apache.org/.

Apache Spark. (2022). Diambil kembali dari https://spark.apache.org/.

Armbrust, M., Huai, Y., Liang, C., Xin, R., & Zaharia, M. (2015, April 13). Deep Dive into Spark SQL’s Catalyst Optimizer. Diambil kembali dari https://databricks.com: https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html

Chanowich, E. d. (2001). Query Optimization Advanced.

Goutam, S. (2021, Februari 12). Apache Spark Logical And Physical Plans. Diambil kembali dari https://blog.clairvoyantsoft.com/: https://blog.clairvoyantsoft.com/spark-logical-and-physical-plans-469a0c061d9e

Han, J. a. (2000). Data Mining Concepts & Techniques. Morgan Kaufmann Publishers.

Harianto Antonio, Novi Safriadi. (2012). Rancang Bangun Sistem Informasi Administrasi Informatika.

Korth, H. d. (1991). Database System Concepts. Singapura: McGraw Hill.

Leturgez, L. (2020, Juli 23). Spark’s Logical and Physical plans … When, Why, How and Beyond. Diambil kembali dari http://www.medium.com: https://medium.com/datalex/sparks-logical-and-physical-plans-when-why-how-and-beyond-8cd1947b605a

Ni Ketut Dewi Ari Jayanti, Ni Kadek Sumiari. (2018). Teori Basis Data. Yogyakarta: Penerbit ANDI.

Rahardja, U. R. (2017). Design of Business Intelligence in Learning Systems Using iLearning Media. Universal Journal of Management, 227-235.

Russell, S. J. (2016). Artificial Intelligence : a modern approach. Malaysia: Pearson Education Limited.

Sunarya, A. S. (2015). Sistem Pakar Untuk Mendiagnosa Gangguan Jaringan Lan. CCIT, 8(2), 1-11.

Wahono, R. S. (2014, Januari 10). romisatriawahono.net/2014/01/10/kontribusi-penelitian-dan-perbaikan-metode/. Diambil kembali dari romisatriawahono.net: https://romisatriawahono.net/2014/01/10/kontribusi-penelitian-dan-perbaikan-metode/

Y. Bengio, A. C. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1798–1828

Downloads

Published

30-06-2022

How to Cite

Dariato, E. (2022). Analisa dan Perancangan Machine Learning Untuk Mendeteksi Kegagalan Job di Apache Spark. Arcitech: Journal of Computer Science and Artificial Intelligence, 2(1), 1–18. https://doi.org/10.29240/arcitech.v2i1.4124

Issue

Vol. 2 No. 1 (2022): June 2022

Section

Articles

Citation Check

License

Authors who publish with Arcitech: Journal of Computer science and Artificial Intelligence agree to the following terms:

  • Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
  • Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
  • Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).