Development of a data analysis platform where data collection and accumulation are generalized

Background to the development of the data analysis platform

TomiiOur company is conducting research and development aimed at advanced data utilization for the implementation of smart maintenance and for diagnostics of the status of power equipment. As part of such efforts, we started collecting data from the sensors installed at the facilities in the Oyama area office and the Hasuda area office. (The data include information such as the insolation and power generation related to solar forecasts, as well as oil temperature and water content in the oil for transformer status diagnostics, among other things.) In the meantime, we considered generalizing the data collection function that had been developed for each service. This was how we kicked off the development project.

Let me briefly explain the developed data analysis platform. This platform comprises terminals collecting the data measured by sensors, such as thermometers, and cloud storage for the accumulated data uploaded from the terminals. In the past, we would develop and incorporate the collection and accumulation function for each service, but the configuration of that function was common even among different services. Although we used to develop the same function every time we created a service, this time, the development of the platform eliminated the need for the work. In addition, the terminals collecting the data are easily introduced because of their independence from specific hardware.

TakadaAt the time of software updates, we used to perform the setup on-site or bring an updated terminal on-site as a replacement. However, we are promoting this development so that we can remotely update the software of the terminals. If it becomes possible to perform the update remotely, that will contribute to providing the service more quickly.
Moreover, when collecting data with a new sensor, we currently adjust the settings manually, which inevitably takes time. In the future, we would like to implement a specification where simply setting the sensor to be used for the type of data will cause the platform to automatically reflect the setting and promptly start collecting data.

Putting together a team for the project

TomiiIn terms of each role in the project, Ms. Okai is assigned to the design, construction, and management of the cloud infrastructure, in addition to the service design and development of the user interface. Mr. Takada is assigned to the design and development of the software for the data collection terminals. Mr. Kita is assigned to tasks to assist both Mr. Okai and Mr. Takada.

As the leader I am assigned to manage the entire project. Since I work in another office than the one assigned to the other members, I usually communicate with them mainly via a chat tool. Thanks to the active comments of the members of the project team, we can accurately understand the status and the progress of the project, which is very helpful. Although our discussions sometimes become too intense, we decided to record important exchanges and issues in the project problem management system, including the background to the occurrence and solutions for later reference.

As for the project, we convene regular weekly meetings for development and operation in which we share information, report on progress, and discuss any technical problems in more depth. Because the members worked with enthusiasm each in their own way, I don’t think I faced many difficulties as a leader. Working together with passionate members, I found greater fulfillment in creating something that can be widely used across the entire company and meets the needs.

OkaiActually, I didn’t have many chances to create an application in the cloud before participating in the current project. I did not have to start from zero because I had an experience in application development; however, there were many cases where new knowledge was needed, so I had to relearn many things. Regarding the issues that I didn’t understand or that I worried about, I asked questions or consulted team members, including Mr. Tomii, to resolve the issues, and I received advice from the contracted cloud company. As a result, I proceeded with the work step by step.

Our team maintained a nice work atmosphere. I was able to promote the development in an environment where we supported each other without hesitation; for example, when I asked a question in the chat, an opinion or response was provided quickly. In particular, I felt secure that if I were to encounter any trouble, I would receive immediate help. In fact, when I had to take time off on the day of the confirmation work on-site because of the sudden illness of my child, another member was willing to substitute for me. It was very helpful.

TakadaI was mainly involved in developing the software for the terminals, and the development tool I used was slightly difficult to work with. Only a minimum level of documents was necessary to create the software, and the documentation was not carefully performed. Assuming that the total is 100, only the first 10 was described in the document, so I had a hard time reading the information and constructing the whole software. I had trouble and groped my way through the process by consistently reading the source codes of open programs, searching for useful development information via the Internet, and actually running the program to gain a better understanding. Since it was difficult but interesting, I felt it was worthwhile. I think the time when a software created through that process operates correctly is a joyful moment for engineers like us.

KitaIn my case, an investigation was equally indispensable for whatever I did as were the cases of Ms. Okai and Mr. Takada. In the event of a problem, I would investigate the cause and solve it. The processes were repeated, which delayed the progress.
For example, when I added a function to the data analysis platform, the usage charge was higher than I anticipated. That was because the portion that was not assumed during the preliminary calculation was charged. I was able to realize the importance of selecting how to run an application in the cloud and how to store the data after understanding the operation of the services to be used. I often broke out in a cold sweat, but I think it was good to identify any problems before providing the product to customers.

Each prospect for the future

KitaIn this project, we actively worked on the automation of environmental creation, which we had not experienced before but was necessary for providing the services. Previously, when developing services in the cloud or updating applications running in the cloud, a complicated environment was constructed by manual intervention. I felt it was fruitful for the spread of the data analysis platform throughout the company that the automation support achieved an “increase in efficiency without manual intervention”. I would like to proceed with the continuous development of the project and further apply the knowledge and experience we gained.

OkaiJust like Mr. Kita, I would like to use what we learned not only in my department but to continuously apply it on a company-wide basis. As Mr. Tomii stated earlier that a record was made in the problem management system, I would like to further raise awareness of the accumulation and sharing of experiences and knowledge and pass them on to younger colleagues.

TakadaI felt again that it was important to share problems and search for solutions as members of a development team. I think it is great to work together with other members of the team to expand the frame of reference, which brings about awareness of other opinions as we seek a solution.
Personally, after engaging in this project, I feel that I have enhanced my skills in the design and development of programs. I would like to apply it other projects in the future.

TomiiI am thinking of spreading the data analysis platform in the company to enter the stage of brushing up based on feedback.
Personally, I expect that appropriate utilization of the data analysis platform may lead to a review of the technology that we researched. I would like to collect and combine the data accumulated by our company to generate synergy and increase the range of utilization.
Eventually, I would like to provide the data analysis platform outside the company. I think the platform has infinite possibilities to solve social problems. As a major theme, I would like to solve the problems of the lack of workers because of the declining birth rate and the aging population and the deterioration of facilities and social infrastructures by providing solutions using IoT or cloud technologies.