SongPlz Bot - Slack and Spotify API What is API API stands for Application Programming Interface. The most common example to describe the API is being a waiter in the restaurant.
Interface between human and object Interface Interface is a program that allows a user to interact computers in person or over a network, which is a software intermediary that allows two applications to talk to each other
Kafka with Docker In this post, we would like to go over how to run Kafka with Docker and Python. Before starting the post, make sure you have installed Docker (Docker hub) on your computer.
Step 1. Docker Image Setup Okay, first, let’s create a directory folder to store docker-compose.yml file. The docker-compose file does not run your code itself.
1 $ mkdir ~/docker-kafka && cd docker-kafka You can pull kafka and zookeeper images by using this docker pull command, more detailed explanation can be found in the following link - kafka and zookeeper from Docker Hub.
SongPlz Bot - Architecture Planning 1. Aim The aim of this SongPlz Bot project (Slack Song Recommendation Bot) is to build a data pipeline to create a chatting bot for song recommendation with Serverless architecture which is a way to develop and run applications and services without managing infrastructure (AWS). If you are still unsure about serverless architecture, please click here.
There are many cloud services available providing FaaS such as AWS Lambda, Azure Functions, and Google Cloud Function.
Udacity Program Kafka (figure) Technologies used Apache Kafka, Kafka Connect, KSQL, Faust Stream Processing, Spark Structured Streaming About The Nanodegree Data Streaming skill was gained to be prepared for the next era of data engineering. Learned how to analyze data in real-time using Apache Kafka and Spark, and build applications to process live insights from data at scale.
Program Details During the program, we completed two courses with two projects.
Udacity Program MLOps I participated Machine Learning DevOps Engineer program in Udacity (Certification).
The program was extraordinary great, providing handful hands-on projects with the detailed and accurate human feedback. MLOps was whole different world and opened my eyes to the world of DevOps and machine learning in production level.
The program is consisted of four projects, and each project covered different technology skills and tools. The most important thing the program emphasized was clean code principes, such as refactoring, handling errors, unit-testing, logging, and addressing.
Data Engineer Roadmap The roadmap aims to give a complete picture of the modern data engineering landscape and serve as a study guide for aspiring data engineers.
Based on the provided Data Engineer Roadmap, I would like to dig into what we need to know to be a data engineer.
Data Engineer Roadmap Section 1. CS fundamentals Basic terminal usage Data structures & algorithms APIs REST Structured vs unstructured data Linux CLI Vim Shell scripting Cronjobs OS How does the computer work?
