Dev Requirements
Prerequisites
Stream Connectors
Java 11
Scala 2.12.11
Apache Flink 1.17.2
Apache Kafka 2.8.1
Batch Connectors
Java 11
Scala 2.12.11
Apache Spark 3.5.1
Apache Kafka 2.8.1
Libraries
Make sure you have the necessary repositories for the development
git clone [email protected]:Sunbird-Obsrv/job-sdk-scala.git
git clone [email protected]:Sunbird-Obsrv/connector-sdk-scala.gitSetup
job-sdk-scala
connector-sdk-scala
Adding Dependencies
Stream Connectors
Add the following to your project's pom.xml file under dependencies
Batch Connectors
Add the following to your project's pom.xml file under dependencies
Prerequisites
Python 3.10 or higher
Kafka 2.8.1
Spark (PySpark) 3.5.1
Required Packages
The obsrv python package is distributed through PyPI repository and can be installed using pip
Using Poetry for Dependency Management
Poetry is a popular tool for dependency management and packaging in Python projects. It streamlines the process of installing and updating project dependencies. To get started with Poetry, first install it using the following command
Once installed, you can create a new Poetry project:
To add dependencies to your project, such as the obsrv package, use:
Poetry automatically creates and manages a virtual environment for your project, ensuring isolated dependencies and compatibility management.