feat: upgrade to sumac

CHANGELOG.md

@@ -19,6 +19,23 @@ instructions, because git commits are used to generate release notes:
 
 <!-- scriv-insert-here -->
 
+<a id='changelog-19.0.0'></a>
+## v19.0.0 (2024-10-23)
+
+- 💥[Feature] Upgrade to Sumac. (by @Faraz32123)
+- [Feature] Add Elasticsearch support in tutor-discovery. As Tutor and Open edX have shifted to Meilisearch, and course-discovery still depends on Elasticsearch, running the Elasticsearch container with tutor-discovery will facilitate smoother operation for the course-discovery service. (by @Faraz32123)
+  - Please see related tutor core PR for context https://github.com/overhangio/tutor/pull/1141
+- 💥 [Deprecation] Drop support for python 3.8 and set Python 3.9 as the minimum supported python version. (by @Faraz32123)
+- 💥[Improvement] Rename Tutor's two branches (by @DawoudSheraz):
+  * Rename **master** to **release**, as this branch runs the latest official Open edX release tag.
+  * Rename **nightly** to **main**, as this branch runs the Open edX master branches, which are the basis for the next Open edX release.
+- [Improvement] Move is_docker_rootless method related to elasticsearch from tutor core to tutor-discovery. (by @Faraz32123)
+- 💥[Feature] Update Course Discovery Image to use Ubuntu 22.04 as base OS. (by @hinakhadim)
+- [Bugfix] Fix catalog_service_user permissions and 403 while fetching pathways (by @dyudyunov)
+- [BugFix] Fix images(media) persistance issue by mounting media directory in volumes through patches. (by @Faraz32123)
+- [Bugfix] Fix legacy warnings during Docker build. (by @regisb)
+
+
 <a id='changelog-18.0.0'></a>
 ## v18.0.0 (2024-05-14)
 

Makefile

@@ -1,10 +1,10 @@
 .DEFAULT_GOAL := help
 .PHONY: docs
-SRC_DIRS = ./tutordiscovery
+SRC_DIRS = ./tutordiscovery ./tests
 BLACK_OPTS = --exclude templates ${SRC_DIRS}
 
 # Warning: These checks are run on every PR.
-test: test-lint test-types test-format  # Run some static checks.
+test: test-lint test-types test-format test-unit  # Run some static checks.
 
 test-format: ## Run code formatting tests.
 	black --check --diff $(BLACK_OPTS)
@@ -15,6 +15,9 @@ test-lint: ## Run code linting tests
 test-types: ## Run type checks.
 	mypy --exclude=templates --ignore-missing-imports --implicit-reexport --strict ${SRC_DIRS}
 
+test-unit: ## Run unit tests
+	python -m unittest discover tests
+
 format: ## Format code automatically.
 	black $(BLACK_OPTS)
 

README.rst

@@ -38,7 +38,7 @@ Operations
 Creating a user
 ~~~~~~~~~~~~~~~
 
-The discovery user interface will be available at http://discovery.local.edly.io for a local test instance,
+The discovery user interface will be available at http://discovery.local.openedx.io for a local test instance,
 and at ``DISCOVERY_HOST`` (by default: http(s)://discovery.<your lms host>) in production. To run
 commands from the UI, a user must be created:
 
@@ -46,7 +46,7 @@ commands from the UI, a user must be created:
 
     tutor local run discovery ./manage.py createsuperuser
 
-Then, you must log in with this user at http://discovery.local.edly.io/admin.
+Then, you must log in with this user at http://discovery.local.openedx.io/admin.
 
 Alternatively, you can log in with oauth2 using a pre-existing user created on the LMS/CMS by accessing
 http(s)://discovery.<your lms host>/login. To do so, the proper domain names must exist and point to
@@ -108,13 +108,13 @@ extra argument to the command. i.e. ``--domain``. While running tutor in product
 
 .. code-block:: bash
 
-    tutor local run lms ./manage.py lms cache_programs --domain="local.edly.io"
+    tutor local run lms ./manage.py lms cache_programs --domain="local.openedx.io"
 
 While running tutor in development mode:
 
 .. code-block:: bash
 
-    tutor dev run lms ./manage.py lms cache_programs --domain="local.edly.io:8000"
+    tutor dev run lms ./manage.py lms cache_programs --domain="local.openedx.io:8000"
 
 This last step should be performed every time you create new or make changes to existing programs.
 
@@ -122,7 +122,7 @@ Show Programs Tab
 ~~~~~~~~~~~~~~~~~
 
 To make the ``Programs`` tab work in the LMS dashboard, users will need to manually create an entry
-in the ``Programs api config`` model in the LMS Admin Panel. Go to http://local.edly.io/admin/programs/programsapiconfig/.
+in the ``Programs api config`` model in the LMS Admin Panel. Go to http://local.openedx.io/admin/programs/programsapiconfig/.
 Add ``Marketing path`` equal to ``/programs`` and enable it. Then Programs tab will be shown on the LMS
 where users can view their registered programs. It will show like in the below picture.
 
@@ -132,7 +132,7 @@ where users can view their registered programs. It will show like in the below p
 In the above image, the user can see explore programs button which is pointing to ``http://localhost:8080/programs`` by default.
 This link does not exist. So, users can change this link to their custom-built marketing site URL to show all programs there.
 This can be done by modifying the ``Site Configurations`` model in the LMS Admin Panel. Go to
-http://local.edly.io/admin/site_configuration/siteconfiguration/. Open the respective LMS site configuration and add the below
+http://local.openedx.io/admin/site_configuration/siteconfiguration/. Open the respective LMS site configuration and add the below
 dictionary in ``site values`` field like the below image:
 
 .. code-block:: python
@@ -169,7 +169,7 @@ in the development container:
 
     tutor dev start --mount /path/to/course-discovery/ discovery
 
-You can then access the development server at http://discovery.local.edly.io:8381. Feel free to add breakpoints
+You can then access the development server at http://discovery.local.openedx.io:8381. Feel free to add breakpoints
 (``import pdb; pdb.set_trace()``) anywhere in your source code to debug your application.
 
 Once a local repository is mounted in the image, you will have to install nodejs dependencies and collect static assets:

changelog.d/20240621_170044_regis.md

@@ -1 +0,0 @@
-- [Bugfix] Fix legacy warnings during Docker build. (by @regisb)

changelog.d/20240626_115731_faraz.maqsood_images_persistance_issue.md

@@ -1 +0,0 @@
-- [BugFix] Fix images(media) persistance issue by mounting media directory in volumes through patches. (by @Faraz32123)

changelog.d/20240710_160247_evgen.dyudyunov_fix_service_user_permissions.md

@@ -1 +0,0 @@
-- [Bugfix] Fix catalog_service_user permissions and 403 while fetching pathways (by @dyudyunov)

changelog.d/20241111_172041_faraz.maqsood_remove_py38_references.md

@@ -1 +0,0 @@
-- 💥 [Deprecation] Drop support for python 3.8 and set Python 3.9 as the minimum supported python version. (by @Faraz32123)

changelog.d/20241119_123431_dawoud.sheraz_branch_rename.md

@@ -1,3 +0,0 @@
-- 💥[Improvement] Rename Tutor's two branches (by @DawoudSheraz):
-  * Rename **master** to **release**, as this branch runs the latest official Open edX release tag.
-  * Rename **nightly** to **main**, as this branch runs the Open edX master branches, which are the basis for the next Open edX release.

setup.py

@@ -34,8 +34,8 @@ setup(
     long_description_content_type="text/x-rst",
     packages=find_packages(exclude=["tests*"]),
     include_package_data=True,
-    install_requires=["tutor>=18.0.0,<19.0.0"],
-    extras_require={"dev": "tutor[dev]>=18.0.0,<19.0.0"},
+    install_requires=["tutor>=19.0.0,<20.0.0"],
+    extras_require={"dev": "tutor[dev]>=19.0.0,<20.0.0"},
     python_requires=">=3.9",
     entry_points={"tutor.plugin.v1": ["discovery = tutordiscovery.plugin"]},
     classifiers=[

tests/test_utils.py

@@ -0,0 +1,26 @@
+import subprocess
+import unittest
+from unittest.mock import MagicMock, patch
+
+from tutordiscovery import utils
+
+
+class UtilsTests(unittest.TestCase):
+    @patch("subprocess.run")
+    def test_is_docker_rootless(self, mock_run: MagicMock) -> None:
+        # Mock rootless `docker info` output
+        utils.is_docker_rootless.cache_clear()
+        mock_run.return_value.stdout = "some prefix\n rootless foo bar".encode("utf-8")
+        self.assertTrue(utils.is_docker_rootless())
+
+        # Mock regular `docker info` output
+        utils.is_docker_rootless.cache_clear()
+        mock_run.return_value.stdout = "some prefix, regular docker".encode("utf-8")
+        self.assertFalse(utils.is_docker_rootless())
+
+    @patch("subprocess.run")
+    def test_is_docker_rootless_podman(self, mock_run: MagicMock) -> None:
+        """Test the `is_docker_rootless` when podman is used or any other error with `docker info`"""
+        utils.is_docker_rootless.cache_clear()
+        mock_run.side_effect = subprocess.CalledProcessError(1, "docker info")
+        self.assertFalse(utils.is_docker_rootless())

tutordiscovery/__about__.py

@@ -1 +1 @@
-__version__ = "18.0.0"
+__version__ = "19.0.0"

tutordiscovery/patches/discovery-common-settings

@@ -0,0 +1 @@
+DISCOVERY_RUN_ELASTICSEARCH = {{ DISCOVERY_RUN_ELASTICSEARCH }}

tutordiscovery/patches/k8s-deployments

@@ -32,3 +32,54 @@ spec:
         - name: settings
           configMap:
             name: discovery-settings
+
+{% if DISCOVERY_RUN_ELASTICSEARCH %}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: elasticsearch
+  labels:
+    app.kubernetes.io/name: elasticsearch
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: elasticsearch
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: elasticsearch
+    spec:
+      securityContext:
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        fsGroupChangePolicy: "OnRootMismatch"
+      containers:
+        - name: elasticsearch
+          image: {{ DISCOVERY_DOCKER_IMAGE_ELASTICSEARCH }}
+          env:
+            - name: cluster.name
+              value: "openedx"
+            - name: bootstrap.memory_lock
+              value: "true"
+            - name: discovery.type
+              value: "single-node"
+            - name: ES_JAVA_OPTS
+              value: "-Xms{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }} -Xmx{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }}"
+            - name: TAKE_FILE_OWNERSHIP
+              value: "1"
+          ports:
+            - containerPort: 9200
+          securityContext:
+            allowPrivilegeEscalation: false
+          volumeMounts:
+            - mountPath: /usr/share/elasticsearch/data
+              name: data
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: elasticsearch
+{% endif %}

tutordiscovery/patches/k8s-jobs

@@ -23,4 +23,3 @@ spec:
         - name: settings
           configMap:
             name: discovery-settings
-

tutordiscovery/patches/k8s-services

@@ -10,3 +10,19 @@ spec:
       protocol: TCP
   selector:
     app.kubernetes.io/name: discovery
+{% if DISCOVERY_RUN_ELASTICSEARCH %}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: elasticsearch
+  labels:
+    app.kubernetes.io/name: elasticsearch
+spec:
+  type: ClusterIP
+  ports:
+    - port: 9200
+      protocol: TCP
+  selector:
+    app.kubernetes.io/name: elasticsearch
+{% endif %}

tutordiscovery/patches/k8s-volumes

@@ -0,0 +1,16 @@
+{% if DISCOVERY_RUN_ELASTICSEARCH %}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: elasticsearch
+  labels:
+    app.kubernetes.io/component: volume
+    app.kubernetes.io/name: elasticsearch
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 2Gi
+{% endif %}

tutordiscovery/patches/local-docker-compose-dev-services

@@ -15,3 +15,12 @@ discovery:
     default:
       aliases:
         - "{{ DISCOVERY_HOST }}"
+
+{% if DISCOVERY_RUN_ELASTICSEARCH and is_docker_rootless() %}
+  elasticsearch:
+    ulimits:
+      memlock:
+        # Fixes error setting rlimits for ready process in rootless docker
+        soft: 0  # zero means "unset" in the memlock context
+        hard: 0
+  {% endif %}

tutordiscovery/patches/local-docker-compose-permissions-command

@@ -0,0 +1 @@
+{% if DISCOVERY_RUN_ELASTICSEARCH %}setowner 1000 /mounts/elasticsearch{% endif %}

tutordiscovery/patches/local-docker-compose-permissions-volumes

@@ -0,0 +1 @@
+{% if DISCOVERY_RUN_ELASTICSEARCH %}- ../../data/elasticsearch:/mounts/elasticsearch{% endif %}

tutordiscovery/patches/local-docker-compose-services

@@ -6,7 +6,28 @@ discovery:
   volumes:
     - ../plugins/discovery/apps/settings/tutor:/openedx/discovery/course_discovery/settings/tutor:ro
     - ../../data/discovery/media:/openedx/discovery/course_discovery/media
+    {% if DISCOVERY_RUN_ELASTICSEARCH %}- ../../data/elasticsearch:/mounts/elasticsearch{% endif %}
   depends_on: 
     - lms
     {% if RUN_MYSQL %}- mysql{% endif %}
-    {% if RUN_ELASTICSEARCH %}- elasticsearch{% endif %}
+    {% if DISCOVERY_RUN_ELASTICSEARCH %}- elasticsearch{% endif %}
+
+{% if DISCOVERY_RUN_ELASTICSEARCH -%}
+  elasticsearch:
+    image: {{ DISCOVERY_DOCKER_IMAGE_ELASTICSEARCH }}
+    environment:
+      - cluster.name=openedx
+      - bootstrap.memory_lock=true
+      - discovery.type=single-node
+      - "ES_JAVA_OPTS=-Xms{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }} -Xmx{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }}"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    restart: unless-stopped
+    user: "1000:1000"
+    volumes:
+      - ../../data/elasticsearch:/usr/share/elasticsearch/data
+    depends_on:
+      - permissions
+{%- endif %}

tutordiscovery/plugin.py

@@ -9,6 +9,7 @@ from tutor import hooks as tutor_hooks
 from tutor.__about__ import __version_suffix__
 
 from .__about__ import __version__
+from .utils import is_docker_rootless
 
 # Handle version suffix in main mode, just like tutor core
 if __version_suffix__:
@@ -36,6 +37,12 @@ config: t.Dict[str, t.Dict[str, t.Any]] = {
         "EXTRA_PIP_REQUIREMENTS": [],
         "REPOSITORY": "https://github.com/openedx/course-discovery.git",
         "REPOSITORY_VERSION": "{{ OPENEDX_COMMON_VERSION }}",
+        "RUN_ELASTICSEARCH": True,
+        "DOCKER_IMAGE_ELASTICSEARCH": "docker.io/elasticsearch:7.17.13",
+        "ELASTICSEARCH_HOST": "elasticsearch",
+        "ELASTICSEARCH_PORT": 9200,
+        "ELASTICSEARCH_SCHEME": "http",
+        "ELASTICSEARCH_HEAP_SIZE": "1g",
     },
     "unique": {
         "MYSQL_PASSWORD": "{{ 8|random_string }}",
@@ -121,6 +128,10 @@ tutor_hooks.Filters.ENV_TEMPLATE_TARGETS.add_items(
         ("discovery/apps", "plugins"),
     ],
 )
+# Template variables
+tutor_hooks.Filters.ENV_TEMPLATE_VARIABLES.add_item(
+    ("is_docker_rootless", is_docker_rootless),
+)
 # Load patches from files
 for path in glob(
     os.path.join(

tutordiscovery/templates/discovery/apps/settings/partials/common.py

@@ -20,8 +20,14 @@ DATABASES = {
     }
 }
 
+DISCOVERY_DOCKER_IMAGE_ELASTICSEARCH = "{{ DISCOVERY_DOCKER_IMAGE_ELASTICSEARCH }}"
+DISCOVERY_ELASTICSEARCH_HOST = "{{ DISCOVERY_ELASTICSEARCH_HOST }}"
+DISCOVERY_ELASTICSEARCH_PORT = "{{ DISCOVERY_ELASTICSEARCH_PORT }}"
+DISCOVERY_ELASTICSEARCH_SCHEME = "{{ DISCOVERY_ELASTICSEARCH_SCHEME }}"
+DISCOVERY_ELASTICSEARCH_HEAP_SIZE = "{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }}"
+
 ELASTICSEARCH_DSL['default'].update({
-    'hosts': "{{ ELASTICSEARCH_SCHEME }}://{{ ELASTICSEARCH_HOST }}:{{ ELASTICSEARCH_PORT }}/"
+    'hosts': "{{ DISCOVERY_ELASTICSEARCH_SCHEME }}://{{ DISCOVERY_ELASTICSEARCH_HOST }}:{{ DISCOVERY_ELASTICSEARCH_PORT }}/"
 })
 
 {% for name, index in DISCOVERY_INDEX_OVERRIDES.items() %}

tutordiscovery/templates/discovery/build/discovery/Dockerfile

@@ -1,6 +1,6 @@
 # syntax=docker/dockerfile:1.4
 ###### Minimal image with base system requirements for most stages
-FROM docker.io/ubuntu:20.04 AS minimal
+FROM docker.io/ubuntu:22.04 AS minimal
 
 ENV DEBIAN_FRONTEND=noninteractive
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
@@ -8,7 +8,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     apt update && \
     apt install -y curl git-core gettext language-pack-en \
     build-essential libcairo2 libffi-dev libmysqlclient-dev libxml2-dev libxslt-dev libjpeg-dev libssl-dev \
-    pkg-config libsqlite3-dev mime-support
+    pkg-config libsqlite3-dev media-types mailcap
 ENV LC_ALL=en_US.UTF-8
 
 ARG APP_USER_ID=1000
@@ -47,10 +47,10 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared pip install \
     # https://pypi.org/project/setuptools/
     # https://pypi.org/project/pip/
     # https://pypi.org/project/wheel/
-    setuptools==69.1.1 pip==24.0 wheel==0.43.0
+    setuptools==75.2.0 pip==24.2 wheel==0.44.0
 
 # Install a recent version of nodejs
-RUN pip install nodeenv==1.8.0
+RUN pip install nodeenv==1.9.1
 # nodejs version picked from https://github.com/openedx/course-discovery/blob/master/Dockerfile
 RUN nodeenv /openedx/nodeenv --node=16.14.2 --prebuilt
 ENV PATH=/openedx/nodeenv/bin:${PATH}
@@ -73,7 +73,7 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared pip install \
     # Use redis as a django cache https://pypi.org/project/django-redis/
     django-redis==5.4.0 \
     # uwsgi server https://pypi.org/project/uWSGI/
-    uwsgi==2.0.24
+    uwsgi==2.0.27
 
 {% if DISCOVERY_ATLAS_PULL %}
 # Pull translations. Support the OEP-58 proposal behind a feature flag until it's fully implemented.
@@ -91,13 +91,13 @@ RUN mkdir course_discovery/media
 # Run production server
 ENV DJANGO_SETTINGS_MODULE=course_discovery.settings.tutor.production
 EXPOSE 8000
-CMD uwsgi \
-    --static-map /static=/openedx/discovery/course_discovery/assets \
-    --static-map /media=/openedx/discovery/course_discovery/media \
-    --http 0.0.0.0:8000 \
-    --thunder-lock \
-    --single-interpreter \
-    --enable-threads \
-    --processes=2 \
-    --buffer-size=8192 \
-    --wsgi-file course_discovery/wsgi.py
+CMD ["uwsgi", \
+    "--static-map", "/static=/openedx/discovery/course_discovery/assets", \
+    "--static-map", "/media=/openedx/discovery/course_discovery/media", \
+    "--http", "0.0.0.0:8000", \
+    "--thunder-lock", \
+    "--single-interpreter", \
+    "--enable-threads", \
+    "--processes=2", \
+    "--buffer-size=8192", \
+    "--wsgi-file", "course_discovery/wsgi.py"]

tutordiscovery/utils.py

@@ -0,0 +1,16 @@
+import subprocess
+from functools import lru_cache
+
+
+@lru_cache(maxsize=None)
+def is_docker_rootless() -> bool:
+    """
+    A helper function to determine if Docker is running in rootless mode.
+
+     - https://docs.docker.com/engine/security/rootless/
+    """
+    try:
+        results = subprocess.run(["docker", "info"], capture_output=True, check=True)
+        return "rootless" in results.stdout.decode()
+    except subprocess.CalledProcessError:
+        return False