Ross Williams
310b4d1242
Add htmltotext extractor
...
Saves HTML text nodes and selected element attributes in
`htmltotext.txt` for each Snapshot. Primarily intended to be used
for search indexing.
2023-10-23 21:42:32 -04:00
Ross Williams
6555719489
Add space after tags when extracting text
...
Add space after any close tag to ensure that
tokens that would be rendered separate in HTML
get extracted as separate tokens in text.
Example:
`<p>First</p><p>Second</p>` --> `First Second`
NOT `FirstSecond`
2023-10-16 09:59:08 -04:00
Ross Williams
d8aa84ac98
Make extracting text for indexing optional
...
Add a configuration option to enable/disable HTML text extraction
for indexing
2023-10-12 13:14:39 -04:00
Ross Williams
b6a20c962a
Extract text from singlefile.html when indexing
...
singlefile.html contains a lot of large strings in the form of `data:`
URLs, which can be unnecessarily stored in full-text indices. Also,
large chunks of JavaScript shouldn't be indexed, either, as they pollute
search results for searches about JS functions, etc.
This commit takes a blanket approach of parsing singlefile.html as it is
read and only outputting text and selected textual attributes (like
`alt`) for indexing.
2023-10-12 13:06:35 -04:00
Nick Sweeting
3093057675
bail out on sonic indexing after 5 errors
2021-04-10 05:18:03 -04:00
Nick Sweeting
32764347ce
add new SEARCH_BACKEND_TIMEOUT config option defaulted to 90sec
2021-04-09 00:27:12 -04:00
Nick Sweeting
21bb84f627
fix lint errors
2021-04-06 02:50:51 -04:00
Nick Sweeting
f67a5a215a
fix readability indexing process and implement a max total character length on indexed content
2021-04-06 02:01:38 -04:00
Nick Sweeting
bd6d9c165b
enforce utf8 on literally all file operations because windows sucks
2021-03-27 01:16:29 -04:00
Nick Sweeting
24e24934f7
add headers.json and fix relative singlefile path resolving for sonic
2021-01-30 21:59:34 -05:00
jdcaballerov
14df0cbb7c
Update sonic.py
...
Sonic buffer accepts 20.000 bytes not unicode characters, since the chunking here is on unicode characters, sending 20.000 characters will overflow sonic's buffer.
UTF-8 can take up to 6 bytes, so sending less than (20.000 / 6) rounded minus should be ok.
2021-01-20 14:51:46 -05:00
Nick Sweeting
326fe69eea
fix lint error
2020-12-12 12:35:32 -05:00
jdcaballerov
9b6afa36a3
Update archivebox/search/backends/ripgrep.py
...
Co-authored-by: Nick Sweeting <git@sweeting.me>
2020-12-12 08:36:08 -05:00
jdcaballerov
aa53f4f088
Update archivebox/search/backends/ripgrep.py
...
Co-authored-by: Nick Sweeting <git@sweeting.me>
2020-12-12 08:36:01 -05:00
jdcaballerov
24d4c44624
Add ripgrep configs
2020-12-12 07:36:31 -05:00
Cristian
e82161a768
refactor: Remove setup_django from search
2020-12-11 16:43:48 -05:00
Nick Sweeting
e90cf05141
fix lint errors
2020-12-11 16:51:11 +02:00
Cristian
9aac09a5e1
feat: Patch setup_django so we can use an inmemory db in specific commands
2020-12-08 18:42:25 -05:00
Cristian
8d22ebf988
feat: Remove walrus operator (we still need to support python3.7)
2020-12-06 12:23:02 -05:00
jdcaballerov
172197ae01
refactor: Remove if LENGTH and use text chunker for every input
2020-12-06 01:14:39 +02:00
jdcaballerov
5a6b814c79
Add exception handling for indexable content reader
2020-12-06 01:14:38 +02:00
JDC
15fbd81480
Change MAX_SONIC_TEXT_LENGTH
2020-12-06 01:14:38 +02:00
JDC
db9c2edccc
Add log print for url indexing
2020-12-06 01:14:38 +02:00
JDC
0acf479b70
Partition long strings in chunks for sonic
2020-12-06 01:14:38 +02:00
JDC
caf4660ac8
Add indexing to update command and utilities
2020-12-06 01:14:37 +02:00
JDC
23a9beb4e0
Add ignored extensions in ripgrep search
2020-12-06 01:13:39 +02:00
JDC
95382b3812
Add ripgrep rg search backend and set as default
2020-12-06 01:13:39 +02:00
JDC
4eeedae815
Exception handling for indexing and searching
2020-12-06 01:13:39 +02:00
JDC
fb67d6684c
fix: Return empty QuerySet instead of list
2020-12-06 01:12:47 +02:00
JDC
823df34080
Use QuerySets for search backend API instead of pks
2020-12-06 01:12:47 +02:00
JDC
f383648ffc
Use a generator for snapshot flush from index
2020-12-06 01:12:47 +02:00
JDC
47daa038eb
Implement flush for search backend after remove command
2020-12-06 01:12:47 +02:00
JDC
c2c01af3ad
Add config for search backend
2020-12-06 01:12:47 +02:00
JDC
5f6673c72c
Implement backend architecture for search engines
2020-12-06 01:12:46 +02:00
JDC
b1f70b2197
Initial implementation
2020-12-06 01:12:45 +02:00
