To address these challenges, we have developed a web-based platform that support any AI models integration, manipulate cloud infrastructure, support numerous data types, autogenerate API
Project Overview
Artificial Intelligence (AI) have been transforming various industries, the AI development and deployment become more challenging, especially for small and medium-sized businesses.
To address these challenges, we have developed a web-based platform that support any
AI models integration, manipulate cloud infrastructure, support numerous data types, auto generate API for any AI models, offer AI as a Service (AaaS) marketplace that provides an affordable price model for customers to use these AI APIs without purchase of copyright.
Our platform is designed to provide a user-friendly and accessible AI development and deployment experience for businesses of all sizes It enables users to upload data, train and evaluate models and deploy them to the cloud Our platform supports all popular AI frameworks, including TensorFlow, PyTorch, and Keras, and can integrate any custom AI models developed by users.
One of key features of our platform is automatic creation of APIs for any AI models Once the model is trained, deployed, the platform automatically create generates APIs depended on developer specification These APIs can be used to integrate the models with any application and service This feature enables users to deploy their models to production without worrying about the technical details of creating APIs.
In addition, although our platform is currently able to support JSON input format, our team aim this platform can support all data types, including structured, semi-structured, and unstructured data Moreover, users can easily upload their data in various formats, including CSV, JSON, and text files, and use them to train and evaluate their models.
Furthermore, our platform provides a flexible and scalable cloud infrastructure for deploy- ing AI models Users can choose from various AWS cloud resources that meet requirement and deploy their models in a few clicks The platform also provides advanced monitoring and logging capabilities, enabling users to track the performance of their models and diagnose any issues that may arise.
Finally, our platform provides an affordable pricing model that does not require the purchase of the copyright Customers can use the APIs at an affordable price without having to pay a large amount of money to purchase the copyright This makes our platform an ideal choice for small and medium-sized businesses that want to leverage the power of AI without breaking the bank.
Objectives and Limitation
Objectives
• Study the principle of microservices architecture and cloud computing, then design the system architecture
• Research Back-end technologies such as OAuth2, NodeJS and API gateway, then imple- ment Back-end services
• Research Frontend technologies such as NextJS, TypeScript and LocalStorage, then im- plement the web application
• Research data storage technologies such as MongoDB or AWS S3
• Research and apply Continuous Integration and Continuous Deployment process
• Integrate, connect the modules of the system
Limitation
According to the capstone project, we set the expected limitation as the following:
• Have not defined, analyzed, and prioritized relevant use cases and identified requirements for any specific careers.
• Only support through web-based platform, not in mobile application
• Do not have mechanism or algorithm for recommending users suitable AI models
• Finish all the endpoints and integrate services together on the Backend side
• Improve User interface and add other flows on the Frontend side
• Finish configuration for cloud and storage, setup CI/CD
• Implement unit testing for Backend and Frontend
Review of existing literature and relevant technologies
The vendor-agnostic EMPAIA platform for integrating AI applications
The vendor-agnostic EMPAIA platform solve the challenge of integration between AI ap- plications and multitude of different non-standardized hardware and software systems currently present in the heterogeneous digital pathology landscape These systems include WSI scanners producing vendor-specific image formats, Image Management Systems (IMS), Vendor NeutralArchives (VNA), DICOM Picture Archiving and Communication Systems (PACS), AnatomicPathology Laboratory Information Systems (APLIS), and digital pathology workstations pro- viding local data processing capabilities, as well as a user interface (UI) for pathologists.
Google Colab
Google Colab is a cloud-based platform provided by Google that allows users to write and run Python code in a web browser, using powerful hardware resources provided by Google. Colab is a free service that provides access to a virtual machine with CPU or GPU processors and a range of pre-installed libraries for machine learning and data analysis.
Google Colab provides a web-based notebook environment based on Jupyter notebooks,which allows users to write and execute Python code, add text and images, and create interactive visualizations Users can collaborate with others by sharing their notebooks or working together in real-time using Google Drive Colab also provides access to a range of data sources, including Google Drive, Google Cloud Storage, and popular datasets like MNIST and CIFAR-10.
One of the key benefits of using Google Colab is its integration with Google’s cloud ser- vices Colab provides easy access to Google Cloud Platform services, such as Google CloudStorage and BigQuery, which makes it easier to manage data and scale machine learning projects Additionally, Colab provides integration with popular machine learning frameworks like TensorFlow and PyTorch, making it a popular choice for data scientists and machine learn- ing researchers.
Microsoft Azure ML Studio
Microsoft Azure ML Studio is a cloud-based platform that allows users to create, train, and deploy machine learning models using a visual interface or Jupyter notebooks The platform provides a drag-and-drop interface for building and deploying predictive models, making it easy for users without programming experience to get started with machine learning.
Azure ML Studio provides a range of pre-built machine learning algorithms and data transformation tools, as well as the ability to use custom code for more advanced modeling It also provides access to a range of data sources, including Azure Blob storage, Azure Data Lake Storage, and other cloud-based and on-premises data sources.
One of the key benefits of using Azure ML Studio is its integration with other Azure ser- vices, such as Azure Databricks and Azure Cognitive Services This allows users to easily inte- grate machine learning models with other cloud-based services to build more complex applica- tions.
Teachable Machine
Teachable Machine is a free, web-based tool developed by Google that allows users to easily create machine learning models without any coding knowledge It uses a simple interface to train models using images, sounds, and other data types, and then exports the trained model as code that can be used in various programming languages.
The Teachable Machine interface provides users with three types of input: images, sounds, and poses Users can either upload their own data or use data provided by the tool to train their models The tool allows users to select the type of model they want to train, such as image clas- sification, object detection, or sound classification Once the user has provided enough training data, the tool uses machine learning algorithms to build a model that can be used to classify new data Users can then export the trained model as code that can be used in their own projects.
One of the key benefits of Teachable Machine is its ease of use The tool is designed for beginners and educators who want to introduce machine learning concepts to students or create simple applications that use machine learning The simple interface and intuitive workflow make it easy for users to get started with machine learning without any prior experience.
Existed projects/tools limitations
The vendor-agnostic EMPAIA platform for integrating AI applications
• Limited career support: Only enable interoperability among different existing digital pathology software systems.
• Release speed: Do not have CI/CD mechanism to meet agile development
• Performance evaluation: lack of AI performance testing
• Network risks: Have not got solutions for cyberattacks, especially DDOS (Distributed
Google Colab
• Limited resources: Although Google Colab provides a good amount of resources for free, such as CPU and GPU processing, there are limitations to the amount of resources available This can limit the size and complexity of the models that can be trained.
• Short session times: Google Colab provides a maximum session time of 12 hours, after which the session is automatically terminated This can be a problem for long-running experiments that require more time than that.
• No persistent storage: The data and code saved in Google Colab is stored temporarily and can be lost when the session ends, making it important to frequently save and backup work.
Microsoft Azure ML Studio
• Complexity: The platform can be overwhelming for beginners due to its many features and options, making it harder to get started and fully utilize its capabilities.
• Cost: While there is a free tier available, the more powerful features and larger resources come at a cost, making it less accessible for users who cannot afford to pay for premium features.
• Learning curve: While Azure ML Studio offers many tools and options for machine learning, it requires some level of prior knowledge and expertise in machine learning, as well as some coding skills.
Teachable Machine
• Limited capabilities: Teachable Machine is designed for creating basic machine learning models and may not be suitable for more complex projects or applications.
• Limited data types: Although Teachable Machine supports images, sounds, and poses as inputs, it does not support other types of data, such as text or video, which can be limiting for certain types of projects.
• Limited control: As a beginner-friendly tool, Teachable Machine provides limited con- trol over the training process, which may not be suitable for users who require more customization and control over their models.
Identification of the gap that the proposed project will fill
• Unlimited Resources: Our AI API services are based on AWS cloud infrastructure Due to scalability capability that AWS offer, it can be considered that this solution offers vir- tually unlimited resources in cloud.
• Customizable Resources: AWS offer a wide range of cloud computing resources that are customizable to meet the specific needs of applications.
• Various Data Type Support: AI API service is aimed to support any data types like text, audio, images, and streaming videos, etc.
• License Fee Share & pay-as-you-go Mechanism: Individuals only need to pay a small amount money for copyright due to sharing and e-commercial mechanism (it means the end-user run/manipulate model, but not be able to know the algorithm inside) Besides,Pay-as-you-go is a pricing model offered by AWS that allow you to pay only for comput- ing resources you use, without the need for long-term-commitments or up-front payments.This model provides greater flexibility and cost-effectiveness, particularly for organiza- tion with variable and unpredictable workloads.
• No Session Expiration: AI API service offer various solutions (execution, run-time, com- pute limitation, etc) that allow users to configure or eliminate session expiration while running your AI model themselves.
• AI as Services: AI API service offer a convenient way to manipulate AI through web
APIs which are suitable for numerous devices like IOTs.
• E-commercial Platform: Enables businesses/developers to sell their AI models or ser- vices online, by providing a set of tools and features to manage online transactions, pay- ments, execution, IDE and customer data.
Microservices
What are microservices and containers?
A microservices architecture splits your application into multiple services that perform fine- grained functions and are part of your application as a whole Each of your microservices will be:
Figure 4: Real captionAn example of microservices architecture 1
1 ByteByteGo What Are Microservices Really All About? (And When Not To Use It) htt ps :{{www.youtube.com{watch?v “ lTAcCNbJ7KE&ab_channel “ ByteByteGo
Monolithic architectures are mostly applied by traditional applications In monolithic archi- tectures, all components and functions are in a single instance.
Figure 5: Monolithic and microservices comparison
Containers are lightweight packages of your application code together with dependencies such as specific versions of programming language runtimes and libraries required to run your software services Docker and Kubernetes are the most popular frameworks to orchestrate mul- tiple containers in enterprise environments Compared to virtual machines (VMs), containers share the operating system kernel instead of having a full copy of it, such as making multipleVMs in a single host Although it’s possible to put your microservices into multiple VMs, you would typically use containers in this case since they take up less space and are faster to boot up.
Why use a microservices architecture?
The microservices architecture was applied to deal with problems of monolithic applica- tions Microservices are widely used and preferable, and some large-scale websites already transformed their software architecture gradually in microservices architecture.
There are numerous advantages of using microservices architecture:
• Developers will work on a smaller codebase compared to a larger one in a monolithic app When components of the application are loosely coupled, developers will easily understand the source code and not slow down development Not to mention, your IDE will be faster if you’re working with a lesser number of lines of code Developers will not need to deal with the complexities and dependencies of functions that can be found in a monolithic app.
• Responsibles of developers will ber more specific and defined A team can be assigned by components or microservices of the app Code reviews will be faster Making updates will be quicker and there would be no need to build and deploy everything compared to a monolithic app.
• Various techology stack can be used The application will no longer need to depend on one language or library Microservices can leverage different programming languages as
1 developer.ibm.com developers see fit It is possible to have polyglot microservices such as the ones.
• Continuous delivery process will be easier.Compared to a monolithic app, you will not need to deploy everything again for a simple change with microservices You may choose to rebuild and deploy the only microservice that needs to be updated Frequent updates will be faster.
• Each services will be scalable individually You can choose to scale each component of your app depending on the resource it needs You wouldn’t need to make multiple instances of everything compared to a monolithic app Scaling the microservices will efficiently use the resources available instead of having multiple copies of the whole application in a monolithic app.
• Data can be decentralized You can choose to use different databases or storage for your microservices You can choose a NoSQL database if your microservice suits it better than a relational database A microservice may also only need a simple key-store database such as Redis As in the diagram below, you can choose a combination of Cloudant, MySQL, and MongoDB databases You can take advantage of different databases to store different data types.
• Isolate failures An error or a bug in one microservice does not bring the entire system down When you have loosely coupled components and a microservice in your application hangs or throws errors, there’s less chance that other microservices will be affected since they are in their own containers and don’t entirely depend on each other A monolithic app can bring the entire process of your application down if the bug or error is not caught properly.
Microservice drawbacks
Switching an application from monolithic architecture to microservices arhchitecture have lots of challenges In generally, those problems are implementation complexity and cost.
• When number of microservices grow, it can be hard to keep track of them Moreover, setting up continuous integration and continuous delivery(CI/CD) can be harder.
• Complexity of co-ordination Microservices require multiple teams to develop and main- tenance of of those The increase of individual services leeds to more network calls in cases where it needs to interact to other microservices.
• Finding and tracing the bugs and errors in application Microservice communicates with multiple other microservices, it can consume lots of time and effort by just looking for that error.
• Routing between microservices will need more work Gateway implementation and mi- croservices flow control are essential Microservices version need to be kept track care- fully in order to configure the routing.
• The consumption of resources are greater than monolithic one Although microser- vices architecture are better choice in scalability, resource usage efficiency It takes more usage of memory and CPU to run separate containers for those microservices.
Docker Compose and Kubernetes
• Docker Compose: is a tool for defining and running multi-container Docker applica- tions It allows developers to define the services that make up an application in a single YAML file, and then start up all the services with a single command Docker Compose is primarily designed for local development and testing, and is often used as a lightweight alternative to Kubernetes With Docker Compose, developers can easily create and man- age multi-container applications on their local machines, without the need for a complex deployment infrastructure.
• Kubernetes: on the other hand, is a much more powerful and complex container orches- tration platform It is designed to manage and deploy containerized applications at scale,with features such as automatic scaling, load balancing, and self-healing Kubernetes pro- vides a way to manage containerized applications across multiple nodes and clusters, and can handle complex application architectures and deployment scenarios Kubernetes is often used in production environments, where high availability and reliability are critical.
WhileDocker ComposeandKuberneteshave different use cases, they can be used together in certain scenarios For example, developers can use Docker Compose to define and manage multi-container applications on their local machines, and then use Kubernetes to deploy those same applications to production environments Additionally, developers can use Kubernetes to manage Docker Compose applications running on a cluster of machines, allowing for a more scalable and powerful development and testing environment.
Cloud Computing
Benefits of cloud computing
• Agility The cloud offers a wide range of services that people can innovate software faster and build nearly any kind of application Cloud providers can help customers deploy technology services in a matter of minutes, and get from idea to implementation several orders of magnitude faster than before.
• Elasticity The property of a cloud to enhance or shrink capacity for CPU, memory, and storage resources to adapt to the changing demands of an organization Cloud Elasticity can be configured automatically, without need to perform capacity planning in advance of the occasion, or it can be a manual process where the organization is notified they are running low on resources and can then decide to add or reduce capacity when needed.
• Cost savings Due to elasticity character, businesses can optimize resource utilization economically The cost of physical operation team and hardware maintenance are reduced considerably Furthermore, the cloud providers have disaster recovery plans which may waste lots of money if IT staff does not understand clearly and commitment of percentage down-time annually can strengthen the customers believes in resilience infrastructure.
Types of cloud computing
• Public cloud It is a type of hosting in which cloud services are delivered over a network for public use Customers do not have any control over the location of the infrastructure. The cost is shared by all users and is either free or in the form of a license policy like pay per user Public clouds are great for organizations that require managing the host application and the various applications users use.
• Private cloud It is a cloud infrastructure that is solely used by one organization It gives organizations greater control over security and data safeguarded by a firewall and man- aged internally It can be hosted internally or externally Private clouds are great for or- ganizations that have high-security demands, high management demands, and uptime re- quirements.
• Hybrid cloud It uses both private and public clouds but can remain separate entities.
Resources are managed and can be provided either internally or by external providers.
A hybrid cloud is great for scalability, flexibility, and security An example of this is an organization that can use the public cloud to interact with customers while keeping their data secured through a private cloud.
• Community cloud It is an infrastructure that is mutually shared between organizations that belong to a particular community The community members generally share similar privacy, performance, and security concerns An example of this is a community cloud at banks, the government in a country, or trading firms A community cloud can be man- aged and hosted internally or by a third-party provider A community cloud is good for organizations that work on joint ventures that need centralized cloud computing ability for managing, building, and executing their projects.
Cloud computing in our project
In this project, we have manipulated an ecosystem of AWS (Amazon Web Service), such as DynamoDB (we will discuss this in section6.2.5), Simple Storage Service (section6.2.6) and Elastic Container Service (ECS)
1 Definition: Amazon Elastic Container Service (ECS) is a fully managed container or- chestration service provided by Amazon Web Services (AWS) It allows you to easily run, scale, and manage Docker containers (section 4.1.4) and applications on a cluster of Amazon EC2 instances.
When a request from the internet comes in, it first comes to Route 53(A DNS Server). All of our application endpoints will be declared in Route53 so that the request will come to the right EC2 instance.
When a request comes from the internet, it first comes to the Application Load Balancer(ALB) (layer 7), ALB will redirect the request to the suitable target group (a group ofEC2 instance) Inside each target group, there will be a auto-scaling group which takes the responsibility of forward the request to the available EC2 instance, if all instances are busy, it will automatically create a new instance to handle the request.
An ECS cluster contains a lot of ECS Container Instance, inside each instance, there are several tasks, they are all docker containers, however, they are not normal container, they are call sidecar container as it contains two images, one for HAProxyand one for the codethat we deploy
3 Sidecar container: As we mention above, a task is also called a sidecar container as it contains two docker images, which are HAProxy (section6.2.2) and the image of our code (main image).
The main image exposes the localhost endpoint through a port (8080 by default) to the HAProxy and HAProxy exposes HTTPS endpoint to Route53 It means that if a request wants to come into the main image, it must travel through HAProxy We are able to con- figure HAProxy to allow or deny some endpoints, manage all request and response of the
1 aws.amazon.com, 2018 main image
Localstack
LocalStack is a software development tool that allows us to create local AWS cloud environ- ments on local environment It provides a fully functional for various AWS services, including S3, Lambda, DynamoDB, and others, enabling us to test their applications locally before de- ploying them to the cloud With LocalStack, we can simulate AWS cloud environments and test their applications in a safe and isolated way, without incurring the cost of using actual AWS re- sources LocalStack is open-source and can be easily installed and configured on a developer’s local machine.
In this project, we have used LocalStack to stimulate S3 and DynamoDB In general, Local- Stack is a Docker image, which can be install by using a Docker file and Docker compose: version: "3" networks: credentials_network: driver: bridge ipam: config:
# gateway: 169.254.170.1 services: localstack: image: localstack/localstack container_name: localstack ports:
- DATA_DIR=/tmp/localstack/data
- DEFAULT_REGION=ap-southeast-2 volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock" networks: credentials_network: ipv4_address: "169.254.170.5"
In this Docker file, we have declared that we used two services, which ares3anddynamodb, the local port for this image is from 4563 to 4599.
CI/CD
Continuous Integration(CI)
CI/CD is a term of frequently deliver applications to customers by using automation into the stages of application development Continuous integration, continuous delivery, and continuous deployment are 3 main concepts of CI/CD CI/CD is a solution to the problems integrating new features changes or hot fix the bugs that can cause for development and operations teams. Specifically, CI/CD introduces ongoing automation and continuous monitoring throughout the lifecycle of applications The flow starts with fintegration and testing phases before going to delivery and deployment Taken together, these connected practices are called as a "CI/CD pipeline" and are supported by development and operations teams working together in an agile way.
Continuous Integration(CI)
In modern application development, developers usually are responsible for some specific features in the application to develop the application simultaneously However, if an organiza- tion is set up to merge all branching source code together on one day (known as “merge day” or the end of a sprint in Scrum methodology), the resulting work can be tedious, manual, and time-consuming Those problems come from conflict of development in isolated developers workspace.
Continuous integration helps developers merge their code changes back to a shared branch, or “trunk,” more frequently—sometimes even daily Once a developer’s changes to an appli- cation are merged, those changes are validated by automatically building the application and
1 redhat.com running different levels of automated testing, typically unit and integration tests, to ensure the changes cannot break the application.
Continuous Delivery & Continuous Deployment(CD)
Continuous Delivery Following the automation of builds and unit and integration testing in CI, continuous delivery automates the release of that validated code to a repository There- fore, in order to have an effective continuous delivery process, it’s important that CI is already built into your development pipeline The goal of continuous delivery is to have a codebase that is always ready for deployment to a production environment.
In continuous delivery, every stage—from the merger of code changes to the delivery of production-ready builds—involves test automation and code release automation At the end of that process, the operations team is able to deploy an application to production, staging, development environments quickly and easily.
Continuous Deployment The final stage of a mature CI/CD pipeline is continuous deploy- ment As an extension of continuous delivery, which automates the release of a production-ready build to a code repository, continuous deployment automates releasing an app to production. Because there is no manual gate at the stage of the pipeline before production, continuous de- ployment relies heavily on well-designed test automation.
In practice, continuous deployment means that a developer’s change to a cloud application could go live within minutes of writing it (assuming it passes automated testing) This makes it much easier to continuously receive and incorporate user feedback Taken together, all of these con- nected CI/CD practices make deployment of an application less risky, whereby it’s easier to release changes to apps in small pieces, rather than all at once There’s also a lot of upfront investment, though, since automated tests will need to be written to accommodate a variety of testing and release stages in the CI/CD pipeline.
Frontend Development
ReactJS
ReactJS is a powerful and flexible library for building user interfaces (UIs) that was created by Facebook and is available for free as an open-source tool The library has gained significant popularity among developers because of its flexibility and efficiency in creating interactive UIs for web and mobile applications The component-based architecture of ReactJS helps developers to create small, reusable UI elements called components that can be combined to form complex applications Those components can be easily shared across different projects or teams, which helps to reduce development time and costs.
Virtual DOM is also a crucial concept when discussing about ReactJS It is an in-memory representation of the browser’s DOM When any of the application’s state changes, React com- pares the previous and current versions of the virtual DOM and only updates the parts of the
UI that have changed This approach leads to faster performance because it avoids the need to re-render the entire page each time there is an update.
Next.JS
Next.JS is a framework created by the developers at Vercel to boost building complex Re- act applications It is open-source, free to use, and has become increasingly popular among developers for its powerful features and ease of use.
One of the most significant features of Next.js is its support for server-side rendering (SSR). SSR is a technique that involves rendering a web page on the server before sending it to the client This approach improves the performance of the application by reducing the amount of work the client’s browser needs to do It also enhances search engine optimization (SEO) be- cause search engines can easily index the content of the page.
Figure 19: Server side rendering (SSR) 1
Next.JS can also benefits users by providing automatic code splitting, which is a technique that optimizes the performance of the application by only loading the necessary code for each page This feature reduces the amount of code that needs to be downloaded, leading to faster load times and a better user experience Another plus of using Next.JS is that it integrates well with other popular web technologies such as TypeScript, GraphQL, and CMS platforms.
Figma
Figma is a cloud-based design tool that enables designers to collaborate on projects in real- time Figma’s collaboration features make it easier for multiple designers to work on the same
1 tothenew.com project simultaneously, facilitating quick iteration and feedback The tool provides vector editing tools, which enable designers to create high-quality designs using shapes, paths, and typography. These tools, combined with Figma’s library of plugins and integrations with other design tools, make it a highly productive environment for designers.
Overall, Figma is a powerful and flexible design tool that simplifies the process of designing high-quality products Its cloud-based approach, collaboration features, vector editing tools, prototyping capabilities, and design library make it a popular choice for designers working on a wide range of projects.
Requirements
Functional requirements
• Support 2 types of end user:CreatorsandConsumers
– Provide a code editor for the creators to implement the algorithm
– Let creators submit the algorithm to generate the API
– Provide an interface for creators to write documentation and define inputs
– Let creators publish the API for free or with pricing
– Let the consumers explore and view the APIs
– Provide options for consumers to subscribe to the API (consumers must subscribe to the API to be able to utilize it)
– After subscribed, provide an interface for consumers to utilize the API
• Validate each requests from users
• Keep track on users’ access token, regenerate refresh token
• Manage access to the API, grant access for only creator and consumers who subscribed Python service module:
• Receive algorithms and models from client side
Non-functional requirements
• The system shall allow accidental downtime for at most 5 minutes
• Server shall support service for 1000 users simultaneously
• The system shall be available on Android and iOS
• Software user interface must be user-friendly, everything can be done in just 3-5 steps
• System should ensures security, especially in authorization and subscription payment
Use-case diagrams
Figure 21: Explore APIs use case diagram
Figure 22: Subscribe to APIs use case diagram
Figure 23: Create API use case diagram
Detailed use-case diagram descriptions
Table 1: Table format of Explore APIs use case
Use case name Explore APIs
Related requirements Users can explore APIs and view their general information
Goal in the context The web application shows users list of APIs containing their general information
Successful end condition The web application shows users list of APIs via many ways
(APIs product page, top subscribed APIs, )
Failed end condition The web application shows an empty notification
Primary actors Users (API creators & API consumers)
Trigger Users go to home page to view the feed or explore page to view list of APIs
1 Users go to home page to view the feed or explore page to view list of APIs
2 The web application shows top APIs or the full list of APIs with sort and filter
Exception 1 The server cannot send the response to the client side
Table 2: Table format of Subscribe to APIs use case
Use case name Subscribe to APIs
Related requirements Users can add APIs to cart and select some of them to sub- scribe
Goal in the context Some APIs is subscribed by the user can be utilized
Preconditions The user is authenticated and authorized APIs are already added to cart by a user and that user has enough balance
Successful end condition Users successfully subscribe to some APIs and can utilize them
Failed end condition The web application shows an error notification
Primary actors Users as API consumers
Trigger Users select some APIs in cart and checkout
1 Users add APIs to cart
2 Users select some of them and checkout, confirm subscript- ion
1 Users don’t have enough balance to afford the subscription or the server fails to send the response to the client side
Table 3: Table format of Create an API use case
Use case name Create an API
Goal in the context An API created and owned by the API creator
Preconditions The user is authenticated and authorized
Successful end condition An API is created and owned by the API creator, other users can view, subscribe to it and utilize
Failed end condition The web application shows an error notification
Primary actors Users as API creators
Trigger Users click on Create API button
1 API creators fill in the form with API name, description a- nd prices
2 API creators provide main logic and upload models and su- bmit
3 API creators provide documentation and inputs required
1 An API creator chooses the API name that he/she already had an API with that name
2 The algorithm provided by the creator is unable to compile
Table 4: Table format of Generate API use case
Use case name Generate API
Related requirements The Backend server will compile the algorithms and generate the API
Goal in the context An utilizable API generated from the Backend server
Preconditions The Backend server must receive the algorithms and/or mod- els from the client side
Successful end condition The Backend server sends success response to client side and the web application lets API creator provides documentation
Failed end condition The Backend server sends error response to client side
Primary actors Backend (Python service)
Trigger API creators submit the algorithms and models using an inter- face of the web application to the Backend server
1 The Backend server receives algorithms from client side
2 The Backend server (Python service) compiles, generates the API
1 The Backend server (Python service) fails to compile the code or fails to send the response to the client side
System design
Our project is an application of the Microservice Architecture The back-end side contains two core services called Smart API service and Authentication Moreover, we are using two more service, MongoDb Atlas cloud and AWS S3.
Smart API and Authentication service communicate with each other by REST API using OAuth2 (Open Authentication 2.0) protocol, which is encrypted by JWT (JSON Web Token).
Our main service is deployed to an EC2 instance which runs in a public subnet in a VPC(Amazon Virtual Private Cloud) The service connects with other AWS services through internet
System Components
Frontend
Our web application is developed with Next.JS and Typescript It communicates with back- end services by REST API with the enhancement from SWR.
Next.JS is a is a Javascript framework built on top of React library which helps developers to create high performance web applications and enhance static websites speed Next.JS is chosen because of its ease of use, fast and high flexibility Some more advantages of using Next.JS are:
• Supporting dynamics imports, which helps keeping Next.JS client-side bundles small. This technique, which is called Code-splitting, allows us to lazily-load application code or dependencies, flexible as required by a specific user decision By this way, JavaScript bundles are divided into smaller chunks (many small JavaScript files).
• Supporting dependency injection (DI), which is very helpful for writing unit testing as it requires developers to inject the dependency in the class constructors (constructor injec- tion) DI makes is easy to maintain and extend the feature as it reduces the dependence between classes It is easy to mock the dependency object to the class which we want to test through constructor.
• Server side rendering (SSR): All the processes are handled on the server side When the client side sends a request to load the page, the server first response with the empty user interface (without any functions or actions) After the procedure of "Hydrating" (when the user interface is integrated with all the JavaScript codes), users can now work with the fully functional web application This technique benefits a lot comparing to the original way (client side rendering) considering its improvement in the load time, the ability to handle higher server load, scalability,
• With respect to SWR, we can take advantages from this technology’s mechanism The idea behind SWR is that every request will be indexed by a identified key or list of keys.
At the first time the request is called, it will store the response to cache (stale) and at anytime the request is called again, it will response immediately with the cached reponse,but along with that, it will also revalidate the reponse from the Backend and then the changes will be merged to the overall response This technology helps us a lot in boosting the performance of the web application and ensures that all the API requests are in the controlled state.
This web frontend will be deployed directly on to the Amazon Web Service (AWS) and run in aEC2 (Elastic Compute Cloud) instance
AWS HAProxy Load Balancer
As we are developing an API service, it is apparent that the number of incoming requests is absolutely high Thus, we need a Load Balancer to maximize the service throughput, increase performance by reducing response time.
HAProxy (High Availability Proxy) is an open source Load Balancer application It is widely used by many big systems which are under numerous access such as Github, Twitter, Reddit, Stackoverflow, HAProxy provides us with a lot of useful functions:
• Load balancing for layer 4 (TCP) and layer 7 (HTTP)
• Support HTTP protocols, HTTP/2, gRPC
There are there algorithms for load balancing:
• Round-robin forwards client requests to each server in a sequential order, and when the algorithm reaches the end of the list of servers, it starts over from the beginning This algorithm is commonly known as round-robin server load balancing This approach works well when the servers being load balanced have similar computing capabilities and storage capacity By distributing the client requests evenly across all servers, round-robin server load balancing helps prevent any one server from becoming overloaded with requests, which can cause slow response times or even downtime.
• Least connectiondistributes client requests to the application server with the least num- ber of active connections at the time the request is received If there are multiple servers with the same lowest number of active connections, then the algorithm follows the round- robin method and assigns the new request to the server that has its turn This approach helps minimize the rate of server overloading and ensures that requests are distributed evenly across all available servers However, this algorithm is more complicated than the default round-robin algorithm and requires more processing power to implement effec- tively.
• Source IP Hash load balancinguses a hashing technique to allocate clients to a specific server Specifically, the algorithm hashes the source and destination IP addresses of the client and server to generate a unique key This key is then used to determine which server should handle the client’s request Since the key can be regenerated if the client’s session is broken, this approach ensures that the client is redirected to the same server if it reconnects to the session This can be useful if it’s important that the client reconnects to an active session after a disconnection and reconnection.
In this project we are using HAProxy, which is provided by AWS and the main algorithm is Round-robin since our service needs a capability of processing several requests and rough computing for machine learning algorithms.
Smart API Service
This is one of the two core services in our microservices architecture It is writen in NodeJSExpress frameworks When a request is sent from client and authenticated by the authentication service, it is forwarded to this service by API Gateway The forwarded request contains the username of the user who sent the request and a token which indicate the session This service is triggered and runs whenever the request is incoming There are several API in this service:
• Create project: this API allows creators to create their own projects (also known as workspace), where their APIs are lying.
• Edit project: this API allow creators to edit their projects’ properties such as: name, alias, and especially project privacy which indicates who can view, edit the project.
• Delete project: this API allows creators to delete their projects If a project is deleted, all API belongs to it are also deleted.
• Create API: this API allows creators to create their own API with python files APIs are located in the certain project where they created.
• Edit API: creators are able to edit their APIs: name, alias, python file, and especially APIs privacy.
• Delete API: creators are able to delete their APIs after creating
• Expose created APIs: comsumers can manipulate the created APIs by sending a
POST request to the endpoint: {v1{creator_username{pro ject_alias{api_alias where creator_usernameis the username of the API creator,pro ject_aliasis the project’s alias of the project which is used to distinguish among projects of a user and contains the proper API andapi_aliasis the api’s alias which is used to distinguish among APIs of a project
The request header contains the username and password of the consumers and the body contains the input in JSON format Otherwise, the error exception is sent back to the sender This API privacy is obviously public or shared with or in other words, the sender must have the permission to access the API
Python service
Since we are working a lot with AI models and algorithm, we need a CPU bound interpreter and Python is the best choice for us because of it multi-threaded interpreter, large amount of pre-build libraries and Python is the most widely used for AI
The exposing created APIs functions contains a python interpreter which is used to run the python file of the API whenever a request from API consumers is coming Besides the error handlers provided by python3 interpreter, we have to implement some more complicated error handlers:
• Infinite loop and infinite recursion: limit the execution time to two seconds (API creator have to pay more for more execution time)
• Prevent bad lines such as deleting directory, files, by disabling the os import
• Limit the library import to reduce the bad code lines
• Disable the account immediately when the user try to inject bad code lines
DynamoDB
Since our services are deployed to Amazon web service, using their own ecosystem is con- sidered to be a good choice, which may reduce the implementation cost, ensures high availability and low latency Thus, DynamoDB is the best option for a database service.
• High scalability: DynamoDB can handle millions of requests per second and can auto- matically scale up or down to meet the demand It can also handle large amounts of data storage.
• High performance: DynamoDB provides fast and predictable performance with low la- tency It uses SSD storage to achieve high read and write throughput, and its data model is optimized for query performance.
• Fully managed: DynamoDB is a fully managed service, which means that AWS takes care of the infrastructure, scaling, and maintenance, allowing developers to focus on building applications.
• Flexible data model: DynamoDB has a flexible data model that allows for easy storage and retrieval of semi-structured data, such as JSON documents It also supports multiple data models, including key-value, document, and graph.
• Highly available and durable: DynamoDB provides a highly available and durable service with automatic multi-region replication and backups It also has built-in fault tolerance
• Cost-effective: DynamoDB offers a pay-per-use pricing model, which means that you only pay for the resources you consume This can be more cost-effective than traditional relational databases, which require upfront investments in hardware and software licenses.
AWS S3
Amazon S3 is an internet storage service that is built to handle large amounts of data and is available across multiple regions It is designed to provide a low-cost solution for storing data, while also ensuring that the stored data is secure, durable, and scalable Due to the large size of the pre-trained AI models that your project works with, AWS S3 is an excellent solution to meet our storage needs.
• Amazon S3 provides a secure storage solution by encrypting data at the object level as it is written to disks in their data centers, and then decrypting it when accessed by users. This process ensures that the data is protected and the user’s license rights are preserved. There are three primary ways that Amazon S3 achieves this level of security:
– Encryption in transit: Amazon S3 encrypts data as it travels between the user and their S3 bucket This helps prevent unauthorized access to the data while it’s being transmitted.
Figure 26: Server-side encryption with Amazon S3 managed keys (SSE-S3) 1
– Server-side encryption: When data is stored in an S3 bucket, it is encrypted at the object level by default This means that even if someone were to gain access to the
1 Viblo.asia, 2018 underlying disk where the data is stored, they would not be able to access the data without the proper decryption key.
– Client-side encryption: Amazon S3 also allows users to encrypt their data using their own encryption keys before uploading it to their S3 bucket This provides an additional layer of security, as the data is encrypted using keys that are only known to the user.
Figure 27: Server-side encryption with customer-provided keys (SSE-C) 2
As our project requirement of storing AI pre-trained model, we decided to choose a stan- dard encrypting method which does not require high level knowledge about AWS and optimize effort: server-side encryption with Amazon S3 managed keys (SSE-S3)
• S3 Creates several of replicas of data and store them at others availability zones to ensure that the data will not be lost in case of disaster.
• S3 Supports version control for all data (like Github)
• S3 always checks the integrity of the data for every data transaction by using checksum.
If the data is lost or broken, the replicas is used for recovery.
• S3 auto re-scale the storage for the amount the data we use Thus, we do not worry about the full storage
Authentication and Authorization
OAuth roles
There are four roles in OAuth2.0 standard:
• Resource owner: in our project, resource owner is the API creator
• Resource server: it is the Python and Nodejs service for create and subscribe api
• Client: is the api subscriber
• Authorization server: is the Auth service written in Nodejs, this service provide autho- rization token for the above roles
Protocol flow
• (C) When a user wants to subscribe to an API, they click on the subscribe button and a subscribe request is sent to the Authorization server, the Authorization server checks for user balance, if the current balance is enough, the authorization server then return an access token for the user (D)
• (E) The user then send the request with the token receive from the Authorization server to the resource server and get the Resource (F)
Note: in our project, there is only one resource server, so that we do not need to implement flow
OAuth grant types
There are several grant types in Oauth2.0 standard: Authorization Code grant type, Client Credentials grant type, Device Code grant type, Refresh Token grant type, Implicit grant type. And we have implemented Authorization Code grant type:
Figure 30: Authorization Code grant type 1
1 Client sends an authentication request, containing their username and password to the Auth service
2 The service checks for the username and password, if they are true, the service returns an access token and a refresh token to the client
3 The client sends other request to others service using this token for authentication
4 The access token will be expired in about 10 minutes, if the client wants to renew the token without sending the username and password again, they can send the refresh token to the Auth service to get a new access token
JSON Web Token (JWT)
The access token that we used in this project is JSON Web Token (JWT).
JWT consists of three parts:
• Header: The type of encoded object in the payload and any extra encoding
This tells us that we have a JWT that is integrity protected with the HMAC SHA-256 algorithm The payload with a JWE including this header will be of a JWT signed and encrypted with the HMAC SHA-256 algorithm.
• Payload: The payload contains the JWT object itself, and the JWT itself is just a set of claims
This tell us that the username of this token is JohnDoe, "iat" is the datetime that this token is issued and "exp" is the expire time
• Signature: This is the most important part of a token that helps the authorization server to check if the token sent from client is valid or not
The main parts are encoded then concatenated with a “.” separating them, so that it looks like:
Example of a token response from authorization server:
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwidXN lcm5hbWUiOiJKb2huRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJleHAiOjE1MTYyNDAwMjJ9.yXOlOJUqtVDU6H4iCcgHlYjf05_EODT1uVtb9s95Dxk"
Development tools and environment
In out project, we have manipulated several tools in order to make it more convenient:
– Visual Studio Code: A powerful code editor for all developers which provides a plenty of plugins, smart suggestions, debugging, auto-complete,
– Postman: A feature-rich platform to work with and test APIs
– Gitlab: A DevOps platform provide the capability of CI/CD implementation and version control
– Figma: A collaborative design tool for creating user interfaces, designs, and proto- types for web and mobile applications
– Jira: A instinctively all-in-one project management tool for tracking and managing tasks, workflows, progress, retro and planning
– Confluence: A proficient collaboration and documentation tool for our members to create and organize notes, documents and information
First glance at the web application
• The web application displays a landing page that can be viewed by anyone
• When user clicks on "Get started", the web application brings user to the home page
• User can browse the APIs without authentication, but to use some functions of the app (creating, adding APIs to cart, utilizing, ), user must register an account and login
Start creating an API
• User goes to the API workspace page
• User can select the option "Start creating" or click on the drop down menu tab "Create API" in the sidebar
• The web application provides a form for user to define the name, description and prices for the API
• The web application then provides a code editor for user to implement the algorithm in Python and upload models
Figure 36: Provide a code editor for creator to implement the algorithm in Python and upload models
• When user clicks on "Submit algorithm" after finish implementation
– If the API successfully generated, the web application provides creator an interface for user to write a documentation and define inputs required for the API
– If failed to generate the API, the web application shows error notification
• After finishing documenting for the API, user clicks on "Submit" to publish the API
Figure 37: Interface for creator to provide documentation and define inputs required
Search for an API
• User can search for the APIs in many ways
1 User is at Home page:
- User can type in the search box in the top bar or search subscribed APIs directly with a small card display at the right side
- The web application brings user to Explore page filtered with the search query
2 User is at Explore page, then user can browse APIs as usual
3 User is at API workspace page:
- User can click on "Search subscribed APIs "
- The web application shows a modal for user to search subscribed APIs
Figure 38: User can search at Home page
Figure 40: User can search at the API workspace page by clicking on "Search subscribed APIs "
Figure 41: The web application shows a modal for consumer to search subscribed APIs
Subscribe and utilize an API
• User (as a creator or consumer) clicks on an API
• The web application brings user to API detail page
• If user hasn’t subscribed to that API:
– The web application shows the message to let user add API to cart
– User can click on "Add to cart" to add API to cart
– If added successfully, the web application shows a success notification
• User can go to Cart page to view APIs added to cart, select some APIs and then choose to subcribe to them or remove them from cart
Figure 42: API detail page for APIs that can be utilized instantly
Figure 43: Subscription payment modal for user to subscribe to an API
• If user is the creator or the consumer who subscribed to that API:
– The web application shows that the API can be utilized
– User can click on "Start using" to start utilizing the API
– The web application shows an interface for user to provide input(s) and observe output
– User fills in the inputs of the table to provide the payload required to utilize the API
– User clicks on the run button to call the API and observe the result
Figure 44: Provide user an interface to utilize when click on
Unit testing
1 Unit testing for Back-end:
We have implemented unit testing for Authentication Service usingJest
Figure 45: Testing coverage detail of Authentication service
We have reached 80%coverage since we focus only on the functions that perform core logic only (e.g controllers, domains, )
2 Unit testing for Front-end
We have implemented unit testing for Frontend side usingJestandReact-testing-library
Figure 48: Testing coverage detail of the web application
Performance testing
We have done performance testing for Authentication service by usingJMeter
We assume that there are 1500 users using the service at the same time, configure the JMeter as follow:
The current configuration of the ECS tasks is: 0.5GB memory, 0.5GHz and 3 concurrent tasks
After running the test, we get the result through Application Load Balancer (ALB) The minimum response time is 0.1s and the maximum response time is 0.18s, which is an acceptable result for the business
Achievements
+ Backend:About the Backend side, we have:
• Implemented APIs for authentication service (authentication, authorization, )
• Implemented CRUD APIs for working with API projects (API service)
• Implemented APIs to check Python source code and generate APIs (Python service)
• Implemented an API Gateway for our microservices
• Written unit testing for Backend with Jest
+ Frontend:On Frontend side, we have:
• Scaffolded a project with TypeScript, Next.JS and SWR
• Implemented an MVP web application, integrated APIs from Backend services (authen- tication service, Python service, )
• Written unit testing for Frontend with Jest and React Testing Library
• Solution design for the whole project using cloud computing
• Use Docker for testing environment
• Suggest CI/CD methods and standards
• Built the whole infrastructure with Terraform
Future Enhancement
Backend
Regarding to the requirements side, we are going to implement these features:
• Comment on API: similar to commenting on some products on the marketplace, this function allows users to put their comments on the API if necessary This must be useful for those who want to report bug, recommend new feature, for the API owner.
• Support more protocols: currently, we have supported HTTP POST request with json body However, the input data of AI is diverse (image, audio, documents, ), which re- quires us to support more protocols for data transfer and even data streaming.
• SmartAPI-SDK:after subscribing an API, user must use some libraries to sendHTTPPOST requests to the server to get the result, which makes it difficult to maintain and the user code and help us to extend supported kinds of request (MQTT, TCP, UDP, ) (related SDK: aws-sdk, adafruit, ).
• Bad code lines detection:allow users to write and submit their codes means that we are going to face with a high risk of security break Preventing users bad submissions (infinite while loop, recursive calls, removing file systems, ) plays an important part in protecting the whole product.
• User environments separation: currently, there is only one environment for users who want to create and sell an API, which make it hard to test since they have to test their API somewhere else before submission We are going to implement two independent environments for user to create, test and publish their API.
• Caching:in most Iot systems, the input data for AI or analysis usually stays unchanged in a period of time (e.g temperature, humidity, ) Thus, we need a caching methodology to reduce response time and save the resource of the server.
Design Pattern
Regarding to the microservices architecture, we are going to implement:
• API gateway:API gateway plays an important part in microservices architecture It acts as a central entry point for all API requests, provides a layer of security by enforcing authentication and authorization mechanisms, offers traffic management capabilities, al- lowing us to control and regulate the flow of API requests and comprehensive monitoring and analytics capabilities.
Summary
The aim of this project is to research and develop an open API platform offering an au- tomatic and implicit AI development and deployment experience, and functioning as a mar- ketplace where users sell, subscribe or utilizing the APIs In this thesis, we have conducted a considerable contribution to the field of software engineering through two key areas: enhancing the process of AI/API development and figure out a solution to develop a favorable product for users to develop, exploring and utilizing the APIs.
The first part of the thesis focused on forming an architecture for our system By conducting researches and hands-on developments, valuable insights were gained regarding microservices architecture, design pattern, database design, Open Authorization2.0 standard, However, it is important to acknowledge that there are aspects within this area that remain unresolved or require further investigation For instance, API gateway, caching methodology, etc
Moving on to the second area explored, the approach empowering the way users can develop, explore and utilizing the APIs The research conducted in this domain helps us to come up with a marketplace where people can create and sell their own products Yet, it is important to note that there are still unexplored aspects within this area For example, cost calculation, user code analytic, etc
Overall, this thesis has introduced an open platform that simplifies and accelerates the AI/API development and deployment process The platform’s marketplace feature serves as a catalyst for innovation, fostering collaboration and knowledge sharing among developers and other domains’ specialists The advancements made during this thesis can hold a significant po- tential to revolutionize the way AI is developed and utilized, empowering a wealthy community of users to leverage the capabilities of AI technologies.
[1] Ben Schwarz (December 2, 2021) "Next.js Performance: Making a Fast Framework
Even Faster" [Online] Available:https://calibreapp.com/blog/nextjs-performance
[2] Mike N¨othiger (October 27, 2020) "How To Set Up a Continuous De- ployment Pipeline with GitLab CI/CD on Ubuntu 18.04" [Online] Avail- able:https://www.digitalocean.com/community/tutorials/how-to-set-up-a-continuous- deployment-pipeline-with-gitlab-ci-cd-on-ubuntu-18-04
[3] Girdharee Saran (February 21, 2019) An Introduction to S3 Server-Side Encryption
(SSE) [Online] Available:https://www.whizlabs.com/blog/s3-server-side-encryption/
[4] Dipanshu Bhatnagar (August 22, 2022) AWS S3 – Client and Server Side Encryp- tion [Online] Available: https://www.encryptionconsulting.com/amazon-s3-simple- storage-service-encryption-at-a-glance/
[5] Webandcrafts (October 15, 2021) Advantages and Disadvantages of
MongoDB NoSQL Database: An Ultimate Guide [Online] Available: https://webandcrafts.com/blog/mongodb-advantages-and-disadvantages/
[6] Dinko Korunic (January 31, 2018) HAProxy on AWS: Best Practices Part 1 [Online].
Available:https://www.haproxy.com/blog/haproxy-amazon-aws-best-practices-part-1/
[7] Anthony Amanse (October 31, 2018) Why should you use microservices and con- tainers? [Online] Available: https://developer.ibm.com/articles/why-should-we-use- microservices-and-containers/
[8] Christoph, Bjorn, Klaus, Daniel, Tobias, Nick, Michael, Lars, Theodore, Andre, Nor- man (March, 2023).The vendor agnostic EMPAIA platform for integrating AI appli- cations into digital pathology infrastructures [Online] accessed April 2023 Available: https://www.researchgate.net/The-vendor-agnostic-EMPAIA-platform
[9] Vercel, Inc "Next.JS documentation" [Online] Available: https://nextjs.org/docs/getting-started
[10] Cristian Vega (February 28, 2017) Client-side vs server-side rendering: why it’s not all black and white [Online] Available: https://www.freecodecamp.org/news/what- exactly-is-client-side-rendering-and-hows-it-different-from-server-side-rendering- bd5c786b340d/
[11] Vercel, Inc "SWR documentation" [Online] Available: https://swr.vercel.app/docs/getting-started
[12] "JWT Encoder" [Online] Available: https://developer.pingidentity.com/en/tools/jwt- encoder.html
[13] (October 2012)."The OAuth 2.0 Authorization Framework" [Online] Available: https://datatracker.ietf.org/doc/html/rfc6749
[14] Kevin Pennekamp (Aug 26, 2019) "How to create a scalable and maintainable front- end architecture" [Online] Available: https://dev.to/kevtiq/how-to-create-a-scalable- and-maintainable-front-end-architecture-4f47
[15] Microsoft "Monaco Editor documentation" [Online] Available: https://microsoft.github.io/monaco-editor/docs.html
[16] Michael Geers "Micro Frontends" [Online] Available:https://micro-frontends.org/
[17] Amazon Web Services, Inc "Class: AWS.S3 - AWS SDK" [Online] Available: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
[18] Amazon Web Services, Inc "Boto3 documentation" [Online] Available: https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
[19] Meta Platforms, Inc "Jest documentation" [Online] Available: https://jestjs.io/docs/getting-started
[20] Kent C Dodds "React Testing Library documentation" [Online] Available: https://testing-library.com/docs/react-testing-library/intro/
[21] OpenAPI Initiative, OpenAPI Specification, 2022,https://spec.openapis.org/ oas
[22] M Jones, J Bradley, N Sakimura, JSON Web Token (JWT), 2015 [Online] Available: