A Troublesome Web Scraping Tale: A Guide to Avoiding "Fake DRM" Video Downloads and Cookie Contamination

In a recent automation project, I needed to batch download over 1,000 course videos from an online education platform (based on Polyv/BokeCC video cloud). I initially thought this would be a routine M3U8 parsing and download task, but I never expected to fall into a time-consuming trap involving "private DRM encryption."

After repeated packet capturing, testing, and refactoring, the real culprit turned out to be an incredibly basic detail—cross-domain Cookie contamination. Here, I'll recap the entire process of hitting pitfalls and finding a way out, hoping it can help developers with similar needs.

The Pitfall Log: Those Seemingly Unsolvable Errors

In the early stages of the project, I had successfully obtained the video's M3U8 playlist and parsed out the .ts segment addresses and keys encrypted with AES-128. However, during the download and merging process, I encountered the following bizarre phenomena:

1. Player error 0xC00D36C4: The downloaded MP4 file size looked normal, but double-clicking it immediately showed a file corruption error with no picture.
2. FFmpeg error Invalid data found when processing input: When trying to use the native FFmpeg command line to merge and decrypt the stream, the underlying process directly refused to handle it.

Wrong troubleshooting direction: These phenomena looked, in terms of technical characteristics, very much like advanced private encryption for streaming media (such as modified byte order, or DRM similar to Widevine). Following this line of thought, I spent a huge amount of time researching CDN defense mechanisms and trying to reverse-engineer the player's JS decryption algorithm, only to sink deeper and deeper.

The Truth Revealed: There Was No Black Magic at All

After a thorough backtrack, the truth was laughable: There was no modified private DRM at all. The standard AES-128 encryption could be decrypted directly. The only reason the video was corrupted was contamination by "poisoned Cookies."

1. The Fatal Cross-Domain Cookie Contamination

In my early request code, I sent the large Cookie containing the main site's login state, along with the request headers, to BokeCC's CDN video segment server. The CDN server has strict domain and credential validation mechanisms. When it received "illegal Cookies" not belonging to its domain, it directly triggered a defense mechanism and sent down contaminated, malformed garbage data. The download tool merged this garbage data into an MP4, so naturally even the file header was wrong.

2. Schizophrenic User-Agent

To bypass the complex PC-side risk control, I disguised myself as a mobile device (iPhone UA) when fetching the API links. However, when initiating the download requests, I forgot to unify the UA and defaulted to the PC-side identifier. The server detected a device jump within the same session, causing some segments to directly return 403 Forbidden.

The Solution: The Ultimate Simple Architecture

The final solution wasn't complex; the core was "channel isolation" and "outsourcing to tools."

Solution 1: API Uses Authentication, CDN Uses a Clean Channel

When calling yt-dlp to actually fetch the video stream, resolutely do not pass the main site's Cookie, only keep the Referer. When the CDN finds no interfering elements, it will obediently send down standard encrypted segments.

cmd = [
    'yt-dlp',
    '--add-header', 'Referer: ',
    '--add-header', 'User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_0...)', 
    # 🚫 Never add Cookie parameters to prevent CDN error interception
    '--concurrent-fragments', '4', # Enable multi-threaded segment download
    '-o', output_filepath, url
]

Solution 2: Globally Unify the Mobile UA

Define the mobile UA as a global constant. Whether requesting playback credentials from the main site API or subsequently fetching the M3U8 list and TS segments, maintain identity consistency throughout, completely eliminating risk control concerns.

Solution 3: Delegate the Dirty Work to Professional Tools

Abandon writing complex FFmpeg pipelines or Python coroutine downloads by hand, and directly call yt-dlp. It can automatically handle M3U8 parsing, concurrent downloads, AES-128 key retrieval, decryption, and final MP4 container packaging, greatly improving code stability and maintainability.

Bonus Tip: How to Prevent Black Screens in Browser Automation?

If you're using browser automation tools like Playwright/Selenium for scraping, you might encounter videos that automatically pause or go black because the page is "in the background" or the window is minimized (triggering the Page Visibility API).

The trick to solving this isn't using headless=True, which is easily detected by risk control, but rather opening a real window and "throwing" it off-screen:

browser = p.chromium.launch(
    headless=False, # Keep headed mode to prevent detection
    args=[
        "--disable-backgrounding-occluded-windows", # 🚫 Prevent black screen for occluded windows
        "--window-position=-2000,-2000",            # 🪄 Physical sleight of hand: throw it to the far top-left corner off-screen
    ]
)

Experience Summary

1. Don't always think "someone is out to get you": When encountering garbled or unparseable data, first check if your own request headers are clean and the protocol is standard. Don't immediately suspect you've hit a world-class DRM problem.
2. Think in terms of variable isolation: The authentication logic for API interfaces and the validation logic for CDN resource distribution nodes are often two separate systems. Don't blindly include all request parameters (especially Cookies).
3. Don't reinvent the wheel: For standard streaming media that yt-dlp can handle, don't write your own TS segment download and AES decryption scripts.