MLReef is the collaboration platform for Machine Learning projects. It is an "end-to-end" development platform that relies heavily on cloud computing for running data pipelines.
The main objective of this task is to enable a per account concurrency in how many pipelines a user can execute at the same time. (e.g. As a user I want to start 10 experiments at the same time). This concurrency is mainly controlled by a. the user membership tier and b. the infrastructure of MLReef.
MLReef has built its services using the CE edition of GitLab. It includes GitLab runners that spawn machines for the data pipelines. These runners are limited in their capabilities (one runner can spawn and manage around 10 machines at the same time). We need a method for scaling GitLab runners based on real-time usage by the users and dependent on the machine type the user selected.
The position required here is DevOps focused (in close work with backend developer), who has experience in:
- AWS Services
- Ideally GitLab Runners (not mandatory, as these are "simple" tasks)
13 freelancers are bidding on average €27/hour for this job
Hi, We are AWS experts and certified and can do this. We can deploy your gitlab runner in either ec2 autoscalling or in kubernetes with cluster scaler . Please get in touch to start. Thanks