We could not create datasets directly from physical tables. Turns out, this is
caused by the fact that reflection is enabled on the clickhouse databases in
Superset. To disable reflection, we manually specify the server version.
In the future, we must remove the server version from the database, and instead
specify engine_reflection=no in the DSN. We will be able to do that once a new
version of clickhouse-sqlalchemy has been released.
See:
https://github.com/xzkostyan/clickhouse-sqlalchemy/issues/151https://github.com/xzkostyan/clickhouse-sqlalchemy/issues/140
This is preventing us from creating datasets from physical tables.
Problem builder units sometimes have subunits with None titles, which causes
the importcoursedata.py script to crash with the following traceback:
Traceback (most recent call last):
File "/openedx/scripts/importcoursedata.py", line 95, in <module>
main()
File "/openedx/scripts/importcoursedata.py", line 32, in main
import_course(course.id)
File "/openedx/scripts/importcoursedata.py", line 40, in import_course
values = [
File "/openedx/scripts/importcoursedata.py", line 41, in <listcomp>
sql_format(
File "/openedx/scripts/importcoursedata.py", line 78, in sql_format
args = [sql_escape_string(arg).decode() for arg in args]
File "/openedx/scripts/importcoursedata.py", line 78, in <listcomp>
args = [sql_escape_string(arg).decode() for arg in args]
TypeError: a bytes-like object is required, not 'NoneType'
See: https://openedx.slack.com/archives/DRW9ECU66/p1628935843000600
See:
Traceback (most recent call last):
File "/openedx/scripts/importcoursedata.py", line 95, in <module>
main()
File "/openedx/scripts/importcoursedata.py", line 32, in main
import_course(course.id)
File "/openedx/scripts/importcoursedata.py", line 66, in import_course
make_query(insert_query)
File "/openedx/scripts/importcoursedata.py", line 88, in make_query
response = requests.post(clickhouse_uri, data=query)
File "/openedx/venv/lib/python3.8/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/openedx/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/openedx/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/openedx/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/openedx/venv/lib/python3.8/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/http/client.py", line 1300, in _send_request
body = _encode(body, 'body')
File "/opt/pyenv/versions/3.8.6/lib/python3.8/http/client.py", line 164, in _encode
raise UnicodeEncodeError(
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f6ab' in position 222493: Body ('🚫') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
Error: Command failed with status 1: docker-compose -f /home/tutor/.local/share/tutor/env/local/docker-compose.yml -f /home/tutor/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/tutor/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm cairn-openedx-job sh -e -c python /openedx/scripts/importcoursedata.py
We sometimes end up with video segments where end_position < start_position,
resulting in negative viewing time. To address this, we discard invalid
segments.
