DevOps: Enable concurrency of Machine Learning pipelines (Kubernetes / AWS / GitLab runners)

MLReef is the collaboration platform for Machine Learning projects. It is an "end-to-end" development platform that relies heavily on cloud computing for running data pipelines.


The main objective of this task is to enable a per account concurrency in how many pipelines a user can execute at the same time. (e.g. As a user I want to start 10 experiments at the same time). This concurrency is mainly controlled by a. the user membership tier and b. the infrastructure of MLReef.

MLReef has built its services using the CE edition of GitLab. It includes GitLab runners that spawn machines for the data pipelines. These runners are limited in their capabilities (one runner can spawn and manage around 10 machines at the same time). We need a method for scaling GitLab runners based on real-time usage by the users and dependent on the machine type the user selected.

The position required here is DevOps focused (in close work with backend developer), who has experience in:

- AWS Services

- Kubernetes

- Ideally GitLab Runners (not mandatory, as these are "simple" tasks)

Skills: Amazon Web Services, Java, Linux, Kubernetes, GitLab

See more: aws machine learning sentiment analysis, aws machine learning, aws machine learning predictive analytics, building machine learning pipelines, aws machine learning tutorial, aws machine learning pricing, aws machine learning documentation, aws machine learning certification, aws vs machine learning, aws machine learning python, aws machine learning projects, aws machine learning university, apply machine learning for devops, how to use machine learning in devops, devops for machine learning, machine learning models kubernetes, deploy machine learning model using aws lambda, deploy serverless machine learning models to aws lambda, deploy serverless machine learning models to aws lambda free download, deploy machine learning model on aws

About the Employer:
( 0 reviews ) Neidling, Austria

Project ID: #32187350

13 freelancers are bidding on average €27/hour for this job


Hi there,I'm biddin on your project "DevOps: Enable concurrency of Machine Learning pipelines (Kubernetes / AWS / GitLab runners)" I have read your project description and i'm an expert in Machine learning/Python/C++/J More

€44 EUR / hour
(23 Reviews)
(7 Reviews)
(14 Reviews)

Hi, We are AWS experts and certified and can do this. We can deploy your gitlab runner in either ec2 autoscalling or in kubernetes with cluster scaler . Please get in touch to start. Thanks

€33 EUR / hour
(52 Reviews)

Hello , I am a developer based out of Hyderabad , India . I have developed and hosted a few applications for my clients on PHP , Laravel , Nodejs , Vuejs , React and React Native . I have also helped my clients More

€20 EUR / hour
(7 Reviews)

☀️☀️☀️☀️☀️ Hello , I hope you are safe and Doing well I have seen your project requirements , I am looking to discuss further with you Hope we will meet soon to discuss further More

€27 EUR / hour
(9 Reviews)

Hi, I am a web designer & developer with 9+ years of experience. I have expertise in Laravel, CodeIgniter, Magento, WordPress, API, CMS, ecommerce, JavaScript, HTML, CSS, MySQL, PHP, Angular, React, React Native, Vue.j More

€22 EUR / hour
(1 Review)

Hi, client. I am Highly interested in your job because I am living on DevOps having 3+ years experiences. Last year I have developed several Devops projects. I can show you those to prove my ability. please send me a m More

€27 EUR / hour
(0 Reviews)

Hello! I can surely help you with end-to-end development with my highest quality performance, having 2+ year of experience as a AWS, ML, Cloud Computing, Kubernetes ,Github, Jenkins, Docker, Linux, CI/CD, and more. More

€20 EUR / hour
(0 Reviews)

Hello, I am Koki based in Japan, Fullstack web developer with over 7 years of hands-on experience in web application development I have read the job description and found it very interesting and I am a good fit for th More

€30 EUR / hour
(0 Reviews)

We have expertise in Azure Architect and AWS, Azure & GCP Certified Professional Cloud Architect with multi-tasking skills having more over 10 years of extensive experience in AWS, Azure & Google Cloud Architecture, Cl More

€18 EUR / hour
(0 Reviews)

Our DevOps expertise: Our DevOps help in transitioning your delivery solutions to the cloud. We Build, test, and deploy applications on the DevOps Cloud with Cloud native tools. We can Dockerize your applications, wri More

€25 EUR / hour
(0 Reviews)

Hi, I have gone through your requirement, and I understood very well that you want Enable concurrency of Machine Learning pipelines. I have more than 4+ years of experience working as DevOps and I have manages more t More

€25 EUR / hour
(0 Reviews)