Dev Requirements

Prerequisites

Stream Connectors

Java 11
Scala 2.12.11
Apache Flink 1.17.2
Apache Kafka 2.8.1

Batch Connectors

Java 11
Scala 2.12.11
Apache Spark 3.5.1
Apache Kafka 2.8.1

Libraries

Make sure you have the necessary repositories for the development

git clone [email protected]:Sunbird-Obsrv/job-sdk-scala.git
git clone [email protected]:Sunbird-Obsrv/connector-sdk-scala.git

Setup

job-sdk-scala

cd job-sdk-scala
mvn clean install

connector-sdk-scala

cd connector-sdk-scala
mvn clean install

Adding Dependencies

Stream Connectors

Add the following to your project's pom.xml file under dependencies

pom.xml

<dependencies>
    ...
        <dependency>
            <groupId>org.sunbird.obsrv.connector</groupId>
            <artifactId>connector-sdk-flink</artifactId>
            <version>1.0.0</version>
        </dependency>
    ...
</dependencies>

Batch Connectors

Add the following to your project's pom.xml file under dependencies

pom.xml

<dependencies>
    ...
        <dependency>
            <groupId>org.sunbird.obsrv.connector</groupId>
            <artifactId>connector-sdk-spark</artifactId>
            <version>1.0.0</version>
        </dependency>
    ...
</dependencies>

Prerequisites

Python 3.10 or higher
Kafka 2.8.1
Spark (PySpark) 3.5.1

Required Packages

The obsrv python package is distributed through PyPI repository and can be installed using pip

pip install "obsrv[batch]"

Using Poetry for Dependency Management

Poetry is recommended for its ease of managing and packaging

Poetry is a popular tool for dependency management and packaging in Python projects. It streamlines the process of installing and updating project dependencies. To get started with Poetry, first install it using the following command

pip install poetry

Once installed, you can create a new Poetry project:

poetry new your_project_name

To add dependencies to your project, such as the obsrv package, use:

poetry add "obsrv[batch]"

Poetry automatically creates and manages a virtual environment for your project, ensuring isolated dependencies and compatibility management.

PreviousObsrv Base Setup NextInterfaces

hashtagPrerequisites

hashtagStream Connectors

hashtagBatch Connectors

hashtagLibraries

hashtagSetup

hashtagAdding Dependencies

hashtagStream Connectors

hashtagBatch Connectors

hashtagPrerequisites

hashtagRequired Packages

hashtagUsing Poetry for Dependency Management

Prerequisites

Stream Connectors

Batch Connectors

Libraries

Setup

Adding Dependencies

Stream Connectors

Batch Connectors

Prerequisites

Required Packages

Using Poetry for Dependency Management