Closed

Local Data Management System

Our company collects various modes of data in the field to conduct condition assessment of critical infrastructure. The data is typically from sensors installed on field robots and includes images, videos, laser scans, environmental data, point measurements etc. The data is also coupled with other miscellaneous attributes such as reports, documents, annotations etc.

The data is currently hosted on local servers and is accessible by the employees on their windows/linux/osx platforms using sshfs, nfs or samba.

However as the amount of data exponentially grows (currently at 30Tb and expecting to double in the next 6 months) we need to address a few challenges in data management.

Current challenges are:

- Data stored across multiple server locations and not accessible seamlessly

- Difficult to prioritise data between ssd and hdd when requiring different speeds for current projects

- No overview of all the data, where it is, when was it last saved etc.

- Data is sometime reshuffled in the back end, which means all the code pointing to hard links breaks.

- Hard to manage permissions which many users trying to do prototyping and development at the same time

- Slow network access on linux machines (could be problem with sshfs)

These problems leads to the basic requirements of the project.

A. Central repository to store all data

A1. A central repository where data can be seamlessly stored. Data is currently spread over a few local machines. This needs to be aggregated such that the end user doesn't have to worry about where the data is actually stored.

A2. Set protocols for adding new data (adding done by plugging in drives into the servers or over the network)

A3. Data access to be streamlined. Data is currently (and will be) stored on local linux servers. Data to be accessible to users positioned on compute servers (connected to monitors and running linux) or their laptops (running linux, windows or OSX). Usable file paths to data for all linux users needs to be consistent to enable code transfer between machines. The same for windows and OSX machines to enable unified file paths when working on data processing software across multiple machines.

A4. Establish correct mounting protocols/scripts needed to allow for maximum read/write speeds.

B A barebones dashboard for the data repository (1 developer, 5-10 days including validation and testing)

B1. View basic stats on all data (where stored, how to access, last edited, size of data, basic notes etc)

B2. Accessible on the network using user login details (if they are an allowed user)

B3. Perform storage medium switches - e.g. copy/move data in the backend between ssd and hdd (for speed boosts on short term projects). This should be seamless to all users, who continue to access the data using the same path.

B4. Ability to change group owner of the different mount points

B5. Ability to see if data is being uploaded to AWS and tag data to be uploaded onto AWS (currently done using cronjob and bash).

Skills: C Programming, C++ Programming, Linux, Network Administration, Python

See more: our company need someone to copy information from some websites into our data management system 20hrweek long term position, our company need someone to copy information from some websites into our data management system 20hr/week long term position , hospital management system demo software, ldm medical abbreviation, ldm data, what is ldm, local data manager, ldm meaning, what is data management system, ldm medicine, linux, web sensor data management system, school management system php software, library management system project software, vtc data management system, pupil data management system, azm web based data management system, student information management system hyderabad software, shoe repair management system free software, php data management system

About the Employer:
( 0 reviews ) Australia

Project ID: #16944203

20 freelancers are bidding on average $4559 for this job

Yknox

Dear,I am Gang Lee . I'm interested in the project you recently posted. I'm a certificated freelancer with over 1000 good reviews from clients. I have experienced knowledge in every part of Web development. I never More

$4444 AUD in 30 days
(622 Reviews)
8.8
hbxfnzwpf

I am very proficient in c and c++. I have 16 years c++ developing experience now, and have worked for more than 7 years. My work is online game developing, and mainly focus on server side, using c++ under Linux environ More

$3000 AUD in 21 days
(128 Reviews)
6.9
$4444 AUD in 30 days
(41 Reviews)
6.6
tudiptechnology

Hi, Greetings for the day, We have been developing/maintaining various web applications in Python Django. Mostly these applications are hosted on clouds (AWS/Linode) and have SASS, CoffeeScript, AngularJS/JQuery, More

$5000 AUD in 60 days
(5 Reviews)
5.9
linuxsupport

I am having an experience of more than 16 years in this field. I have served as a VP-Technology for on of the biggest B2B company of India. I am an expert of AWS and Linux. I have worked on similar type of project befo More

$5555 AUD in 30 days
(60 Reviews)
6.0
schoudhary1553

Hello, I can help with you in your project Local Data Management System. I have more than 5 years of experience in C Programming, C++ Programming, Linux, Network Administration, Python. We have worked on several sim More

$5000 AUD in 30 days
(19 Reviews)
5.0
whizwigmarvel

Dear sir. I'm a senior linux developer from Hong Kong. Please see my profile. I've read carefully the description of project, and i can do it. Let's discuss more details of project. I'll do my best for long-term More

$4150 AUD in 30 days
(4 Reviews)
4.6
$4444 AUD in 30 days
(4 Reviews)
4.6
tekzee

Hi ****We dont do Code in Python, we can only assist you with the DB and AWS**** I just have been through your requirement of Local DB Management and AWS assignments and I found myself fit for this job As we are a More

$4705 AUD in 30 days
(15 Reviews)
4.1
$4444 AUD in 30 days
(7 Reviews)
3.7
CrazyWebGuru

Hello, I had been involved in a project for implementing a large scale relational database management system. So I have a good command of concepts and knowledge of database internals. I am interested in your projec More

$5000 AUD in 30 days
(6 Reviews)
3.8
AhteshamAK

It is my great honor to apply for this vacant job. I am very hardworking and can work effectively as fast as I can. I have excellent skills in VB, VBA, C++, C#, OPPs, HTML5, CSS3, JS, AJAX, JQYERY, React.js, ANGULAR.JS More

$3529 AUD in 30 days
(3 Reviews)
3.3
itoasis1

Hi. I have review your project description. I have full experiences in Data Management Development. I am very interested in your project and also ready to start work immediately. Hoping for your soon Reply. Thank More

$3000 AUD in 30 days
(9 Reviews)
3.3
lavizsolutions

Hi, Kindly share me more details so that i can give you estimated time and cost. Please check my below work.. 1. Case Management system. [login to view URL] you can crea More

$8888 AUD in 65 days
(1 Review)
3.3
MetaoriginLab

Our dedicated team of software and programming professionals provides our clients with state of the art programming solutions in the python language. Me and my team has 5 years of experience into Python/Django,Selenium More

$5023 AUD in 8 days
(7 Reviews)
2.9
TrueGardener

Hello. Thanks for your kind posting. I am good at C Programming, anyway at Data structuring. So i can help you. Let us discuss more in details by chat, Thanks King Regards

$3111 AUD in 30 days
(4 Reviews)
2.7
donnyforu

Hello, I have more than 23 years of experience in top IT companies based in the USA. I have worked for some of the top IT clients across the globe and I have managed and worked on some of the large and complex large More

$4666 AUD in 20 days
(1 Review)
2.2
BEAS24X7

Hi, This is from BEAS Consultancy and Services Pvt Ltd ( [login to view URL] ). We are an India based CMMI level 3 and ISO Certified  Organization. We produce quality and Cost effective solution to our int More

$3333 AUD in 30 days
(1 Review)
1.8
workspaceit

Hello, I m Tahsinul Alam, completed MSC in Soft Engineering now working as one of the project manager in WEB development team of Workspace Infotech Ltd, software/Outsourcing firm located in Dhaka, Bangladesh. We ha More

$4444 AUD in 30 days
(0 Reviews)
0.0
eclipsetechno

Hi, After reading your description, following conclusions could be developed: 1) We shall introduce Mongo DB to store and distribute big data even if it crosses millions. 2) You already suggested AWS, so Clo More

$5000 AUD in 60 days
(0 Reviews)
0.0