Commit graph

3136 commits

Author SHA1 Message Date
Nick Sweeting
a680724367
Merge branch 'dev' into search_index_extract_html_text 2023-10-27 23:09:28 -07:00
Nick Sweeting
720061185c
Merge pull request #1195 from overhacked/method_allow_deny 2023-10-27 23:05:30 -07:00
Nick Sweeting
7a80b61bea
Merge pull request #1252 from c01o/patch-1 2023-10-25 03:28:42 -07:00
c01o
08cffb8742
Fix docker build
Lack of `fi` resulted in a syntax error.
Also, change `;` to `||` to ensure it builds successfully on architectures other than `linux/arm/v7`.


```
[2023-10-24T10:46:49.941Z] ------
 > [dev_container_auto_added_stage_label 11/18] RUN mkdir -p "/home/archivebox/.config/chromium/Crash Reports/pending/"     && chown -R archivebox "/home/archivebox/.config"     ; if [[ "linux/amd64" == "linux/arm/v7" ]]; then $exit 0; else exit 1:
0.401 /bin/bash: -c: line 2: syntax error: unexpected end of file
------
WARNING: buildx: git was not found in the system. Current commit information was not captured by the build
[2023-10-24T10:46:49.942Z] Dockerfile-with-features:135
--------------------
 134 |         && ln -s "$CHROME_BINARY" /usr/bin/chromium-browser
 135 | >>> RUN mkdir -p "/home/${ARCHIVEBOX_USER}/.config/chromium/Crash Reports/pending/" \
 136 | >>>     && chown -R $ARCHIVEBOX_USER "/home/${ARCHIVEBOX_USER}/.config" \
 137 | >>>     ; if [[ "$TARGETPLATFORM" == "linux/arm/v7" ]]; then $exit 0; else exit 1
 138 |         # ignore failure for architectures where no playwright release is available yet
--------------------
ERROR: failed to solve: process "/bin/bash -c mkdir -p \"/home/${ARCHIVEBOX_USER}/.config/chromium/Crash Reports/pending/\"     && chown -R $ARCHIVEBOX_USER \"/home/${ARCHIVEBOX_USER}/.config\"     ; if [[ \"$TARGETPLATFORM\" == \"linux/arm/v7\" ]]; then $exit 0; else exit 1" did not complete successfully: exit code: 2
```
2023-10-24 19:58:23 +09:00
Ross Williams
310b4d1242 Add htmltotext extractor
Saves HTML text nodes and selected element attributes in
`htmltotext.txt` for each Snapshot. Primarily intended to be used
for search indexing.
2023-10-23 21:42:32 -04:00
Nick Sweeting
2681f752f1
ignore playwright chrome install on arm v7 2023-10-21 14:17:50 -07:00
Nick Sweeting
2928c19119 fix multiarch chromium install 2023-10-20 05:45:08 -07:00
Nick Sweeting
988b6585c9 better multiarch builds 2023-10-20 05:30:45 -07:00
Nick Sweeting
25d2c17bb1 add dev build helper script 2023-10-20 05:10:18 -07:00
Nick Sweeting
015737b1e4 fix buildx helper script 2023-10-20 05:10:03 -07:00
Nick Sweeting
bf7a6e4c09 add missing cpython packages needed for arm v8 docker build 2023-10-20 04:33:26 -07:00
Nick Sweeting
63ad43f46c
Merge branch 'dev' into method_allow_deny 2023-10-20 04:25:44 -07:00
Nick Sweeting
a58535baff
Merge pull request #1249 from ArchiveBox/pdm 2023-10-20 04:23:25 -07:00
Nick Sweeting
82d8662c74 add more readability error output 2023-10-20 04:14:28 -07:00
Nick Sweeting
f3e81c1487 bump npm lockfile 2023-10-20 04:09:03 -07:00
Nick Sweeting
76f9b91ed3 dockerfile fixes 2023-10-20 04:08:38 -07:00
Nick Sweeting
6736e63ef0 new Dockerfile layout with better layering 2023-10-20 02:47:34 -07:00
Nick Sweeting
86c662421b show FS_USER in version output debug string 2023-10-20 02:47:23 -07:00
Nick Sweeting
7910a5b34b fix pretty_path printing logic 2023-10-20 02:47:03 -07:00
Nick Sweeting
d4ca2d1154 update mercury bin path to postlight parser 2023-10-20 02:46:41 -07:00
Nick Sweeting
75eeb12ebe dont install youtubedl anymore 2023-10-20 02:46:09 -07:00
Nick Sweeting
16796a63fd share PUID and PGID with child procs from entrypoint 2023-10-20 02:45:54 -07:00
Nick Sweeting
e0e34e6377 update package.json and npm lockfile 2023-10-20 02:45:37 -07:00
Nick Sweeting
3b3bdab97d update ignore files 2023-10-20 02:45:20 -07:00
Nick Sweeting
63c276a93d redirect add page back to snapshots list automatically 2023-10-19 18:24:19 -07:00
Nick Sweeting
22bcffe5ee remove accidental duplicate template code 2023-10-19 18:24:05 -07:00
Nick Sweeting
85f8583d62 catch ldap configuration and packaging errors and make them non fatal 2023-10-19 18:05:59 -07:00
Nick Sweeting
53cff45ec0 switch from pipenv and setup.py to PDM for python packaging 2023-10-19 17:49:06 -07:00
Nick Sweeting
8caffc4b4e ignore .pdm-python file 2023-10-19 17:09:48 -07:00
Nick Sweeting
b6113cc00c use pypa build instead of setup.py build and move ldap to extras 2023-10-19 16:44:48 -07:00
Nick Sweeting
4b26ab9d1f only load image into docker when building, and push on deploy 2023-10-19 16:29:51 -07:00
Nick Sweeting
9fa210515e
Bump required python version to 3.9 in stdeb.cfg 2023-10-19 16:22:06 -07:00
Nick Sweeting
748363e685
Bump required Python version to 3.9 2023-10-19 16:21:32 -07:00
Nick Sweeting
6619e8005e
Update pip.yml github action python version 2023-10-19 16:20:43 -07:00
Nick Sweeting
a61f544547
Update SECURITY.md 2023-10-19 16:18:32 -07:00
Nick Sweeting
edde40898b
fix markdown underlining everything as link 2023-10-19 12:58:38 -07:00
Nick Sweeting
a8ce927937
fix markdown formatting 2023-10-19 12:57:05 -07:00
Nick Sweeting
0e3475a0eb
Create SECURITY.md policy 2023-10-18 18:40:58 -07:00
Nick Sweeting
4f655fc4a1
Replace chown of entire data directory when ownership mismatch detected 2023-10-18 18:24:43 -07:00
Nick Sweeting
e9c85aa812
Update README.md 2023-10-18 17:53:49 -07:00
Nick Sweeting
6f4a7e4e68
Add ADMIN_USERNAME and ADMIN_PASSWORD to example options 2023-10-18 11:53:13 -07:00
Nick Sweeting
2a1104c76a
Merge pull request #1245 from benmuth/add-user-env-vars 2023-10-18 11:50:58 -07:00
Nick Sweeting
d286dca925
better to log before doing a thing than after 2023-10-18 11:47:55 -07:00
Ben Muthalaly
9e6a87114b Fix formatting, logging, logic issues 2023-10-18 12:07:36 -05:00
Ben Muthalaly
521ea70e0c Add check for existing user, change varable names 2023-10-18 03:07:54 -05:00
Ross Williams
6555719489 Add space after tags when extracting text
Add space after any close tag to ensure that
tokens that would be rendered separate in HTML
get extracted as separate tokens in text.

Example:

`<p>First</p><p>Second</p>` --> `First Second`
NOT `FirstSecond`
2023-10-16 09:59:08 -04:00
Ben Muthalaly
44a94157be Remove logging of configured username and password 2023-10-15 23:36:47 -05:00
Ben Muthalaly
11d473e536 Add config options to add admin user on first run 2023-10-14 00:38:04 -05:00
Ross Williams
d8aa84ac98 Make extracting text for indexing optional
Add a configuration option to enable/disable HTML text extraction
for indexing
2023-10-12 13:14:39 -04:00
Ross Williams
b6a20c962a Extract text from singlefile.html when indexing
singlefile.html contains a lot of large strings in the form of `data:`
URLs, which can be unnecessarily stored in full-text indices. Also,
large chunks of JavaScript shouldn't be indexed, either, as they pollute
search results for searches about JS functions, etc.

This commit takes a blanket approach of parsing singlefile.html as it is
read and only outputting text and selected textual attributes (like
`alt`) for indexing.
2023-10-12 13:06:35 -04:00